Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docgen: implement doc link resolution in current module #18642

Merged
merged 4 commits into from
Oct 28, 2021

Conversation

a-mr
Copy link
Contributor

@a-mr a-mr commented Aug 3, 2021

First part of nim-lang/RFCs#125, smart doc links. Only local links (inside a given current module) are supported here.

It works by adding additional anchors from docgen.nim.

An example (modified from nim-lang/RFCs#125):

type SortOrder* = enum k0, k1
proc sortOrder*() = discard
proc foo*(a: int, sortOrder: SortOrder) = discard
proc foo*[T](a: int, sortOrder: SortOrder, c: T) = discard
proc foo*() =
  ## See: SortOrder_ (will link to type)
  ## See: sortOrder_ (will link to proc)
  ## See: foo_ (will link to the whole group of procs 'foo')
  ## See: `foo(int, SortOrder, T)`_ (will link to 2nd overload)
  ## See: `foo()`_ (will link to itself)                                                                                

image

Implementation throws warnings in case of ambiguities. In this case one needs to specify the reference more, e.g. `iterator split`_ or `proc split`_ instead of just split_.

A syntax note:

  • one-word reference can be just suffixed with _ without using enclosing backticks `
  • multiple-words references should be with backticks: `...`_

TODO:

  • make it more WYSIWYG: 1) references like foo_ and `proc foo`_ should display as foo proc 2) `foo(par1, par2)`_ as foo (par1, par2) proc 3) `foo(Type1, Type2)`_ as foo (Type1, Type2) proc
  • when there is only 1 overload: show full name with function parameters (even if it's short-style link like foo_) UPDATE: only from tooltip on hover
  • when there is more than 1 overload for short-style links: show some descriptive link like proc foo (2 overloads)
  • add more tests outlined by @timotheecour
  • add test with inclusion of a module
  • check it in Latex backend
  • check some edge cases like the order dependent example from RFC: currently not working:
    overloads with type constraints on generic parameters

cc @timotheecour

@timotheecour
Copy link
Member

timotheecour commented Aug 5, 2021

Excellent!
seems to work well and is intuitive. After playing with it a bit I found this bug (reduced from a weird behavior i was seeing after modifying compileOption links):

bug 1

when defined case1: # D20210808T110300
  proc foo*(option: string): bool =
    ## * `foo <#foo,string,string>`_ (broken link, not expected to work)

  proc foo2*(option, arg: string): bool =
    ## * foo_ (BUG: doesn't show link to `foo`)

  proc foo3*(option: string): bool =
    ## * `foo3 <#foo,string,string>`_ (BUG2: doesn't link)

  proc foo3*(option, arg: string): bool =
    ## * foo3_ (BUG3: doesn't link)

bug 2

when defined case2: # D20210808T110332
  proc compileOption5*(option: string): bool {.
    magic: "CompileOption", noSideEffect.} =
    ## See also:
    ## * `compileOption5 <#compileOption5,string,string>`_ for enum options

  proc compileOption5*(option, arg: string): bool {.
    magic: "CompileOptionArg", noSideEffect.} =
    ## See also:
    ## * compileOption5_ (BUG: should link to group)

bug 3

when defined case3: # D20210808T110352
  proc foo*(option: string): bool =
    ## See also:
    ## * `foo_ <#foo,string,string>`_  (BUG: shows as ``foo_``)

  proc foo*(option, arg: string): bool =
    ## See also:
    ## * foo_ (BUG: should link to group)

they're probably all related

@@ -819,6 +821,18 @@ proc complexName(k: TSymKind, n: PNode, baseName: string): string =
result.add(defaultParamSeparator)
result.add(params)

proc rstLinkName(k: TSymKind, n: PNode, baseName: string): string =
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

links in lib/system_overview.rst (included from system.nim) work as intended, which is great; can you add a test to ensure links in includes keep working?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, done, thank you for the idea!

@@ -819,6 +821,18 @@ proc complexName(k: TSymKind, n: PNode, baseName: string): string =
result.add(defaultParamSeparator)
result.add(params)

proc rstLinkName(k: TSymKind, n: PNode, baseName: string): string =
## Creates a link anchor in the format compatible with the corresponding
## human-readable RST references.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add tests with operators? eg:

## see `[]`_
## see `[]=`_

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, tests added.

proc rstLinkName(k: TSymKind, n: PNode, baseName: string): string =
## Creates a link anchor in the format compatible with the corresponding
## human-readable RST references.
result = baseName
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add tests with comma separated links, eg:

## see fn1_, fn2_, `fn3`_, `fn4`_

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

lib/packages/docutils/rst.nim Outdated Show resolved Hide resolved
@a-mr
Copy link
Contributor Author

a-mr commented Aug 8, 2021

@timotheecour

In all those bugs you hit a (mis)feature of RST. A link like `foo <site#anchor>`_ generates a link alias foo. When you use foo_ it selects this manually set variant because it has priority. So this is as expected. In the new version a warning is emitted about this conflict with line info where all these variants were originally defined. May be I need to go even further and add to the warnings a phrase that these link aliases were implicitly generated.

Also note that handling of these links `foo <...>`_ were not changed in this PR.
Generally speaking, they are manual and not checked and when you are using them "you are on your own". After this PR and its follow-ups we are not supposed to even use these manual links for linking to symbols, though they will be still relevant for referencing external sites and whole Nim modules (though we can add syntax & automatic resolution even to the latter case at some stage too).

@a-mr a-mr changed the title docgen: implement doc link resolution in current module WIP: docgen: implement doc link resolution in current module Aug 8, 2021
@a-mr
Copy link
Contributor Author

a-mr commented Aug 8, 2021

Marked as WIP as it's a big change and there is no need to rush.

Current list of TODOs:

  • when there is only 1 overload: Don't generate groups (it's better to generate groups anyway to avoid breaking external links if e.g. number of overloads goes 2 -> 1) and show full name with function parameters (even if it's short-style link like foo_)
  • when there is more than 1 overload for short-style links: show some descriptive link like proc foo (2 overloads)
  • add more tests outlined by @timotheecour
  • check it in Latex backend
  • check some edge cases like the order dependent example from RFC:
    currently not working
      ## ref fn_ (OK) and `fn(T)`_ (COLLISION) and `fn(T_2)`_ (NOT FOUND)
      proc fn*[T: SomeFloat](a: T)=discard
      proc fn*[T: SomeInteger](a: T)=discard

@timotheecour
Copy link
Member

timotheecour commented Aug 9, 2021

n all those bugs you hit a (mis)feature of RST. A link like foo <site#anchor>_ generates a link alias foo. When you use foo_ it selects this manually set variant because it has priority. So this is as expected

I see, i thought it would only specify the display text as foo, didn't realize it'd also create a link alias foo; is that something we want to preserve? but fine, if we stick to the "new form" this isn't a problem (but it is a problem if mixing new form and old form as shown in #18642 (comment)

What does it mean for existing moduels that use the existing syntax foo_ <#foo,string,string>_ ? can we upgrade them 1 by one to use instead foo_ or we also need to make sure other modules referencing it use foo_?

In the new version a warning is emitted about this conflict with line info where all these variants were originally defined.

it helps, thanks

May be I need to go even further and add to the warnings a phrase that these link aliases were implicitly generated.

maybe

@a-mr
Copy link
Contributor Author

a-mr commented Aug 9, 2021

What does it mean for existing moduels that use the existing syntax foo_ <#foo,string,string>_ ? can we upgrade them 1 by one to use instead foo_ or we also need to make sure other modules referencing it use foo_?

Yes, we can and actually updating a module as whole eliminates a possibility of collision. Also warning will necessarily be generated so it will be impossibly to forget anything. And besides current links are almost always input as `parseCmdLine proc <#parseCmdLine,string>`_, w.r.t. to our new syntax it has a reverse of parseCmdLine and proc hence there is no collision.

And problems when referencing between modules are impossible because the format of anchors has not changed. All in all, there is almost no risk in this regard.

@Varriount
Copy link
Contributor

Just want to say this is very impressive! Thanks for your work!

@@ -115,3 +115,8 @@ proc f*(x: G[int]) =
proc f*(x: G[string]) =
## See also `f(G[int])`_.
discard

## Ref. `[]`_ is the same as `proc \`[]\`(G[T])`_ because there are no
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would this work to avoid quoting?

## See `proc \`$\`(int)`_
## See ``proc `$`(int)``_

the 2nd is more readable because it's easier to copy paste from a declaration

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not now :-(

In original RST it would not even require escaping, but our implementation was tweaked in favor of Markdown-like inline markup rules, which allow to put the ending backtick anywhere and also put starting backtick anywhere. So with single-backtick syntax we are bound to escaping of backticks.

Regarding double-backticks: the syntax is sane and I like it but it's not standard and we do not support it, though it would probably be easy to implement. I think anything like this will be left for future PRs anyway to avoid conflation here.

I agree it's ugly, that's why to make it less painful I added possibility to ref. operators the short way `$`_ (instead of a long way like

`\`$\``_

)
but for any compound anchors one still has to use escaping:

`proc \`$\``_


## Ref. `[]`_ is the same as `proc \`[]\`(G[T])`_ because there are no
## overloads.

Copy link
Member

@timotheecour timotheecour Aug 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add more tests especially for edge cases, eg:

## See `$`_, `[]=`_
## See `'big`_ (for user defined literals)
## See `foo_bar_` (for `foo_bar`)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added. Thanks for testcases, they helped to catch some bugs.

@Varriount
Copy link
Contributor

Is this still a WIP, or is it ready for a full review?

@a-mr
Copy link
Contributor Author

a-mr commented Aug 13, 2021

Is this still a WIP, or is it ready for a full review?

WIP.

The implementation can still change significantly to add more flexibility ("WYSIWIG") and fix a bug with type constraints.
Also the current scheme of adding aliases may cause memory consumption, I need to do a simple benchmark at least.

@Varriount Varriount marked this pull request as draft August 13, 2021 20:17
@Varriount
Copy link
Contributor

Ok, I've converted this PR to a draft - when it's ready for a full review, you can convert it back and modify the title to remove the "WIP".

doc/docgen.rst Outdated
===================================================

You can reference Nim identifiers from Nim documentation comments (currently
only inside one ``.nim`` file).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe also mention that it works for nim files including an rst file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, mentioned.


A. non-qualified::

foo_
Copy link
Member

@timotheecour timotheecour Aug 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe mention this is the preferred form because:

  • otherwise ppl will get confused as to which form to use by default
  • simpler = better
  • more robust to API changes (eg proc => template, adding optional params etc)

(EDIT: i see you mention it later, but might as well mention it here too)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think one suggestion is enough. Besides some people may like to provide parameters to procs — I would prefer that in most cases also.

doc/docgen.rst Outdated
However for fully-qualified reference copy-pasting backticks (`) into other
backticks will not work in our RST parser (because we use Markdown-like
inline markup rules). In the case of `$` it's allowed to delete backticks,
but in case of `[]` one needs to keep backticks and escape them with
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's a bit nasty, can we avoid this?
this should work and not be ambiguous:

`func []`_ or, in detail:
`func [][T](x: openArray[T]): T`_

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, at first glance I don't see any principal obstacle. It may well be a bug in my working copy :-)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's working now.

doc/docgen.rst Outdated Show resolved Hide resolved
doc/docgen.rst Outdated Show resolved Hide resolved
If you use a short form and have an ambiguity then just add some
additional info.
Brevity is better for reading! If you use a short form and have an
ambiguity problem (see below) then just add some additional info.
Copy link
Member

@timotheecour timotheecour Aug 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about adding something like this:

Furthermore, the simplest form (binarySearch_) is usually preferable because it will generate links that remain valid even if underlying API is modified (e.g. via added optional params, or changing the routine kind).

doc/docgen.rst Outdated
`binarySearch(a: openArray[T], key: K, cmp: proc(T, K))`_
`binarySearch(openArray[T], K, proc(T, K))`_
`binarySearch(a, key, cmp)`_
2. default values in routine parameters are not recognized, one need to
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need => needs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed.

doc/docgen.rst Outdated
`binarySearch(openArray[T], K, proc(T, K))`_
`binarySearch(a, key, cmp)`_
2. default values in routine parameters are not recognized, one need to
specify the type instead. E.g. for referencing `proc f(x = 7)` use::
Copy link
Member

@timotheecour timotheecour Aug 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a TODO for future work somewhere? fixing this would be a nice improvement

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the current implementation does not use Nim compiler facilities to parse all these links, so to fix that for all possible cases would be impossible. The only solution would be to re-write this implementation which is not in my plans.

no backticks: `func $`_
escaped: `func \`$\``_
no backticks: `func [][T](x: openArray[T]): T`_
escaped: `func \`[]\`[T](x: openArray[T]): T`_
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need both? honest question; i feel like just using the no backticks form without giving an option would be best, and simplest to use.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, we do need both. One can want for the link to look syntactically correct (e.g. for educational reasons).

Copy link
Member

@timotheecour timotheecour Aug 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the ideal link would be a copy-paste of the declaration, possible non-ambiguous ways could be:

## see ` func `$`(a: int): string `_
## see ``func `$`(a: int): string``_
## see ```func `$`(a: int): string```_

(at expense of rst spec, which we already violate)

then we'd need only 1 way, and avoid having to give a choice with 2 sub-optimal ways

Copy link
Contributor Author

@a-mr a-mr Sep 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, it's roughly the same as nim-lang/RFCs#355.

this syntax is also not perfect, it does not allow to start from a backtick. While our current syntax does:

`\`$\`(a: int)`_

(Correction: one can input

`` `$`(a: int) ``_

though, it's just that further tweaking of RST inline markup rules in favor of Markdown is needed hear.)

But I agree that such syntax would be good... It's just not among first priorities...

@a-mr a-mr marked this pull request as ready for review September 1, 2021 22:21
@a-mr a-mr changed the title WIP: docgen: implement doc link resolution in current module docgen: implement doc link resolution in current module Sep 1, 2021
@a-mr a-mr force-pushed the smart-links-local-resolution branch from a6a07b9 to fcd064b Compare September 10, 2021 16:34
no special handling of routineKinds here

fix segfault

fix references in the Manual

a supplemental test

generics: rm type parameters in normalized names

allow underscore in names and export (*)

many fixes of warnings:

- always resolve all the 3 cases:
  manual references, RST anchors, Nim anchors and print
  warnings in case of ambiguity
- preserve anchor/substitution definition places
- correctly handle `include` in ``.nim`` files for doc comment
  warnings/errors
- make warning messages more detailed

change wording & add test regarding brackets in input parameter types

some progress:

- fixes for handling symbols with backticks
- now reference points to concrete function (not group) if there is no
  other overloads
- link text is the full name (with function parameters) if there is
  only 1 overload. If there are > 1 then it's like
  "proc binarySearch (2 overloads)"

add more testcases, fix func

add specification of the feature

[skip ci] update the spec

reworked the implementation + more tests

fix ambiguity handling and reporting,

including adding RST explicit hyperlinks to ambiguity printing

Default priorities now conform to the `docgen.rst` spec also.

enable `doc2tex`

fix generics parameters for types

some cleanups and clarifications

tests on inclusion of .rst & .nim into .nim

update forgotten files for tests
@a-mr a-mr force-pushed the smart-links-local-resolution branch from 9acfa1f to 014df3e Compare October 20, 2021 21:48
@a-mr
Copy link
Contributor Author

a-mr commented Oct 25, 2021

@Araq @timotheecour @narimiran

What are you plans for review & conclusion?
Do you have any time during a month? I want to continue working on docgen but this PR kind of hinders me because quite a bit of refactoring was done here, it would be unpleasant to merge that afterwards. If you want we can have a remote session e.g. by Skype discussing this PR.

The current implementation fixes all the problems that were discovered.
It's different from the original code. Original one made a string normalization for both generated anchor and reference and did a simple hash table search. The new one parses a reference ( in dochelpers.nim) so that all parameters (including generics) can be compared with an anchor. Originally I thought about re-using parser code from the compiler but abandoned this idea because:

  • only signatures need to be supported in rst.nim
  • also signatures that don't abide by Nim rules are supported, e.g. one can write walkDir(d) iterator — without type and with iterator at the end. This originated from the existing practice of naming links in Nim's repo.

@Araq
Copy link
Member

Araq commented Oct 26, 2021

Sorry for the delay, we're reviewing it this week.

@narimiran
Copy link
Member

It is hard to do a detailed review of a large PR, but I have tested the changes proposed in this PR by building the docs for strutils (while changing the links to the new short form) and I like the new behavior, e.g. grouping of split funcs when you link to (one of) them.

I'm merging this now; we have plenty of time until next stable version (1.8.0) to iron out any quirks/bugs that might come up which we haven't considered now.

@narimiran narimiran merged commit 7ba2659 into nim-lang:devel Oct 28, 2021
@a-mr
Copy link
Contributor Author

a-mr commented Oct 28, 2021

OK, thx, so I will try to convert all links e.g. in os.nim.

Also the next part of work will be cross-module link support. I hope it will be ready by 1.8.0 also.

Clyybber pushed a commit to Clyybber/nimskull that referenced this pull request Feb 25, 2022
Concerns these changes:
nim-lang/Nim@727c637...340b5a1

Excluded changes are:
nim-lang/Nim#18963
nim-lang/Nim#19003
nim-lang/Nim#19043
nim-lang/Nim#19055
nim-lang/Nim#19053
nim-lang/Nim#19064
nim-lang/Nim#18642
nim-lang/Nim#19062
nim-lang/Nim#19082
nim-lang/Nim#19090
nim-lang/Nim#19077
nim-lang/Nim#19021
nim-lang/Nim#19100
nim-lang/Nim#19102
nim-lang/Nim#19111
nim-lang/Nim#19115
nim-lang/Nim#19133
nim-lang/Nim#19142
nim-lang/Nim#19158
nim-lang/Nim#19129
nim-lang/Nim#19137
nim-lang/Nim#19168
nim-lang/Nim#19156
nim-lang/Nim#19147
nim-lang/Nim#19180
nim-lang/Nim#19183
nim-lang/Nim#19182
nim-lang/Nim#19187
nim-lang/Nim#19179
nim-lang/Nim#19209
nim-lang/Nim#19210
nim-lang/Nim#19207
nim-lang/Nim#19219
nim-lang/Nim#19195
nim-lang/Nim#19212
nim-lang/Nim#19134
nim-lang/Nim#19235
nim-lang/Nim#19252
nim-lang/Nim#19196
nim-lang/Nim#19295
nim-lang/Nim#19301
nim-lang/Nim#19181
nim-lang/Nim#17223
nim-lang/Nim#19370
nim-lang/Nim#19385
nim-lang/Nim#19307
nim-lang/Nim#19394
nim-lang/Nim#19399
nim-lang/Nim#19390
nim-lang/Nim#19407
nim-lang/Nim#19419
nim-lang/Nim#19421
nim-lang/Nim#19363
nim-lang/Nim#19406
nim-lang/Nim#19431
nim-lang/Nim#19455
nim-lang/Nim#19461
nim-lang/Nim@cb894c7
nim-lang/Nim#19462
nim-lang/Nim#19442
nim-lang/Nim#19437
nim-lang/Nim#19433
nim-lang/Nim#19512
nim-lang/Nim#19487
nim-lang/Nim#19543

Excluded changes include major changes which require more consideration
and changes which don't apply to the current code anymore but could be
worth porting over still.

Excluded changes which only change the identifier casing in tests
or only concern code removed in nimskull aren't listed.

Begin commit listing:

use two underscores for easy demangling [backport:1.6] (#19028)

Add Elbrus 2000 architecture (#19024)

* Add Elbrus 2000 architecture

* Add e2k to niminst

* Update compiler/installer.ini

Co-authored-by: Andreas Rumpf <[email protected]>

remove exception (#18906)

allow converting static vars to `openArray` (#19035)

When assigning constant output to a seq, and then passing that static
seq to other functions that take `openArray`, the compiler may end up
producing errors, as it does not know how to convert `static[seq[T]]`
to `openArray[T]`. By ignoring the `static` wrapper on the type for
the purpose of determining data memory location and length, this gets
resolved cleanly. Unfortunately, it is relatively tricky to come up
with a minimal example, as there are followup problems from the failing
conversion, e.g., this may lead to `internal error: inconsistent
environment type`, instead of the relevant `openArrayLoc` error message.

use the correct header for TIOCGWINSZ on Solaris (#19037)

Minor update to terminal docs (#19056)

* Update terminal.nim

- Added some extra docs to cursorUp/Down/Forward/Backward
- I was able to use hideCursor and showCursor without adding stdout, removed the parameter
- Added docs to terminalHeight()* and terminalWidth()*

* Update lib/pure/terminal.nim

Co-authored-by: konsumlamm <[email protected]>

* Update lib/pure/terminal.nim

Co-authored-by: konsumlamm <[email protected]>

* Added back f: file to cursor movement

* Removed unnecessary comments

Co-authored-by: konsumlamm <[email protected]>

fix a tiny formating issue in doc/destructors.rst (#19058)

fix a tiny code snippet formatting issue in `doc/constructors.rst`, again (#19065)

Fix nimIdentNormalize, fixes #19067 (#19068)

* Make nimIdentNormalize return "" when passed ""; fixes #19067

Fixes #19067

* Add tests for nimIdentNormalize

fix #18971 (#19070) [backport:1.6]

since the example code return value from global variable, instead
of first argument, the `n.len` is 1 which causes compiler crashes.

fixes #19000 (#19032)

* fixes #19000

* progress

fix #18410 (Errors initializing an object of RootObj with the C++ backend) [backport] (#18836)

* fix #18410

* one line comment

* typo

* typo

* cover cpp

update numbers of lifetime-tracking hooks in doc/destructors.rst (#19088)

bootstrapping Nim compiler with `cpp --gc:orc` (#19087)

libs/impore/re: Add note about the requirement of `matches` to be pre-allocated (#19081)

Add few runnableExamples for `findBounds` for clarity.

Fixes nim-lang/Nim#18775

Add test for issue 15435 (#19079)

* Add test for issue 15435

Closes nim-lang/Nim#15435.

* Specify bug # in comment

Addresses nim-lang/Nim#19079 (comment)

manual: Document the use of `static` as a proc call (#19084)

* manual: Document the use of `static` as a proc call

Also adds tests.

Fixes nim-lang/Nim#16987 .

* Update doc/manual.rst

Co-authored-by: konsumlamm <[email protected]>

* Use the "bug #NNNN" comment syntax for consistency

Ref:
https://nim-lang.github.io/Nim/contributing.html#writing-tests-stdlib

> Always refer to a GitHub issue using the following exact syntax: bug
for tooling.

* manual: Undocument usage of foo.static

foo.static and foo.static() are not expected to work.

Ref: https://github.com/nim-lang/Nim/pull/19084/files#r741203578

Co-authored-by: konsumlamm <[email protected]>

manual: Document that comma propagates the default values of parameters (#19080)

* manual: Document that comma propagates the default values of parameters

Fixes nim-lang/Nim#15949.

* Use the "bug #NNNN" comment syntax for consistency

Ref:
https://nim-lang.github.io/Nim/contributing.html#writing-tests-stdlib

> Always refer to a GitHub issue using the following exact syntax: bug
for tooling.

fixes #19011 [backport:1.6] (#19114)

Add deprecation pragmas in lib/deprecated/pure (#19113)

Deprecate `std/sharedlist` and `std/sharedtables` (#19112)

fix nimindexterm in rst2tex/doc2tex [backport] (#19106)

* fix nimindexterm (rst2tex/doc2tex) [backport]

* Add support for indexing in rst

Call {.cursor.} a pragma. (#19116)

* Call {.cursor.} a pragma.

Its hard to find .curser annotation while googling because all other things like it are called pragmas. See https://nim-lang.org/docs/manual.html#pragmas
Also the . in front of the name makes it hard to find and search for.

Can we just call it cursor pragma?

* Small fix for comment.

Remove tlsEmulation enabled from Windows + GCC config (#19119) [backport:1.6]

This flag has a very significant performance impact on programs compiled with --threads:on. It is also apparently not needed anymore for standard circumstances. Can we remove the config? See nim-lang/Nim#18146 (comment) for discussion and perf impact. [backport:1.6]

Add security tip for setCookie (#19117)

* Add security tip for setCookie

* Update lib/pure/cookies.nim

Co-authored-by: Dominik Picheta <[email protected]>

* Update lib/pure/cookies.nim

Co-authored-by: konsumlamm <[email protected]>

Co-authored-by: Andreas Rumpf <[email protected]>
Co-authored-by: Dominik Picheta <[email protected]>
Co-authored-by: konsumlamm <[email protected]>

correct cookie docs (#19122)

refactoring: orc can use getThreadId() (#19123)

* refactoring: orc can use getThreadId()

* progress

fixed colorNames sorting mistake (#19125) [backport]

update manual (#19130) [backport]

Merge file size fields correctly on Windows (#19141)

* Merge file size fields correctly on Windows

Merge file size fields correctly on Windows

- Merge the two 32-bit file size fields from `BY_HANDLE_FILE_INFORMATION` correctly in `rawToFormalFileInfo`.
- Fixes #19135

* Update os.nim

Fix punycode.decode function (#19136)

* Refactor: rename proc to func

* Fix punycode.decode function

This function could only properly decode punycodes containing a single
encoded unicode character. As soon as there was more than one punycode
character group to decode it produced invalid output - the number of
characters was correct, but their position was not.

* Update tpunycode.nim

Co-authored-by: Clay Sweetser <[email protected]>

Fix undeclared 'SYS_getrandom' on emscripten (#19144)

wrong spaces (3 => 2) (#19145)

`caseStmtMacros` no longer experimental, experimental manual refactor (#19173)

* `caseStmtMacros` no longer experimental, experimental manual refactor

* Update doc/manual.rst

* apply review suggestions

* apply review

Co-authored-by: Andreas Rumpf <[email protected]>

fix inline syntax highlighting in system.nim (#19184)

swap port to correct port order (#19177)

Co-authored-by: Jaremy Creechley <[email protected]>

feat: TLS-ALPN wrappers for OpenSSL (#19202)

Co-authored-by: Iced Quinn <[email protected]>

misc bugfixes [backport:1.2] (#19203)

treat do with pragmas but no parens as proc (#19191)

fixes #19188

[format minor] remove unnecessary spaces (#19216)

Making TCC work again on Windows --cpu:amd64 - fix #16326 (#19221)

* fix #16326

* removing comments

fixes a converter handling regression that caused private converters to leak into client modules; fixes #19213; [backport:1.6] (#19229)

Add support for LoongArch (#19223)

* Add support for LoongArch

* Update compiler/installer.ini

Co-authored-by: Andreas Rumpf <[email protected]>

remove `std/sharedstrings` (#19228)

* remove std/sharedstrings

it has been broken since 0.18.0

* rephrase the changelog entry

add comments to spawn and pinnedSpawn (#19230)

`spawn` uses `nimSpawn3` internally and `pinnedSpawn` uses `nimSpawn4` internally. I comment it in order to help contributors get the gist of its functionality.

fixes an old ARC bug: the produced copy/sink operations don't copy the hidden type field for objects with enabled inheritance; fixes #19205 [backport:1.6] (#19232)

nimRawSetjmp: support Windows (#19197)

* nimRawSetjmp: support Windows

Using `_setjmp()` directly is required to avoid some rare (but very
annoying) exception-related stack corruption leading to segfaults on
Windows, with Mingw-w64 and SEH.
More details: status-im/nimbus-eth2#3121

Also add "nimBuiltinSetjmp" - mostly for benchmarking.

* fix for Apple's Clang++

Revert "swap port to correct port order (#19177)" (#19234)

This reverts commit 0d0c249.

move toDeque to after addLast (#19233) [backport:1.0]

Changes the order of procs definitions in order to avoid calling an undefined proc.

let Nim support Nimble 0.14 with lock-file support [backport:1.6] (#19236)

nimc.rst: fix table markup (#19239)

Various std net improvements (#19132)

* Variant of  that works with raw IpAddresses.

- Add doc tests for new net proc's.
- Aadd recvFrom impl
- Add recvFrom impl -- tweak handling data var

- Update lib/pure/net.nim
	Co-authored-by: Dominik Picheta <[email protected]>

- cleaning up sendTo args
- remove extra connect test
- cleaning up sendTo args
- fix inet_ntop test
- fix test failing - byte len

* fix test failing - byte len

* debugging odd windows build failure

* debugging odd windows build failure

* more experiments to figure out the windows failure

* try manual assigment on InAddr

Co-authored-by: Jaremy Creechley <[email protected]>

fix bug #14468 zero-width split (#19248)

basicopt.txt: Unify the format (#19251)

fix: fixes bug in CVerifyPeerUseEnvVars (#19247)

Previously CVerifyPeerUseEnvVars was not being passed into
scanSslCertificates, which meant that we weren't scanning
additional certificate locations given via the SSL_CERT_FILE and
SSL_CERT_DIR environment variables

suggestion to respect typedarray type (#19257)

* suggestion to respect typedarray

* Update jssys.nim

Co-authored-by: Sven Keller <[email protected]>

fix #19244 - solves the problem of the InAddr object constructor in Windows. (#19259)

* Update winlean.nim

* Update tnet_ll.nim

Fixed typo in manual.rst unsafeAssign->uncheckedAssign. Fixes part 1 of #19266 (#19267)

use uppercase "type" for Proxy-Authorization header (#19273)

Some servers will reject authorization requests with a lowercase "basic" type. Changing to "Basic" seems to solve these issues.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization

Update colors.nim (#19274)

* Update colors.nim

Added `lightgray` alias to `lightgrey` and `...grey`aliases for the rest of the gray colors.
Added color `rebeccapurple`.
Fixed the incorrect values for the `PaleVioletRed` and `MediumPurple` colors.
This module should now be matching the CSS colors.
I used the seq[tuple] syntax for defining the names.

* Document colors changes.

Extract runnables that specify `doccmd` (#19275) [backport:1.6]

Fix build on FreeBSD/powerpc (#19282)

It's currently misdetected as powerpc64.

Fix #19107 (#19286) [backport]

fixes grammar typos [backport] (#19289)

fix 19292 (#19293)

Fix #19297 - fixing broken list after adding empty list (#19299)

* Update lists.nim

* Update tlists.nim

* removed check `if b.tail != nil`

The tail of the list being null it is still possible to retrieve its end by going through all nodes from the head. So checking for null from `b.tail` is unnecessary. However, setting `a.tail = b.tail` only if `a.head != nil`, so you don't break a good list with an already broken one.

fixes #16617 [backport] (#19300)

Update JS and nimscript import tests (#19306)

* add new modules, except experimental ones
* remove deprecated modules mersenne and sharedlist
* better describe why some modules fail and some modules don't

add compile time option for POSIX sigwait on Illumos/Solaris (#19296)

* add compile time option for POSIX sigwait on Illumos/Solaris

* fix link to documentation of `sigwait` on Illumos/Solaris

[docs] clarify the raised exception (#19308)

* [docs] clarify the raised exception

Lest developers wanna know what the exception is.

* Apply suggestions from @konsumlamm

Co-authored-by: konsumlamm <[email protected]>

Co-authored-by: konsumlamm <[email protected]>

docs: Fix typo in tut1.rst (#19309)

Fix #19314 - fixing broken `DoublyLinkedList` after adding empty `DoublyLinkedList` (#19315) [backport]

* Update lists.nim

* Update tlists.nim

fixed typos (#19316)

devel: style fix (#19318)

this allows "--styleCheck:usages --styleCheck:error"

docs: Fix typo in tut1.rst (#19324)

correct the comments (#19322)

--expandArc

```
var
  a
  b
a = matrix(5, 5, 1.0)
b = matrix(5, 5, 2.0)
`=sink`(b, -
  let blitTmp = b
  wasMoved(b)
  blitTmp +
    a)
`=destroy`(b)
`=destroy`(a)
```

add std/private/win_getsysteminfo; refactor the usage of `GetSystemInfo` (#19310)

* add std/private/win_getsysteminfo

* import at the top level

* wrappers follow nep1 too

* follow review comment

Update net.nim (#19327) [backport]

Fix #19038 - making the Nim compiler work again on Windows XP (#19331)

* Update osenv.nim

* Update win_setenv.nim

* Update lib/pure/includes/osenv.nim

* Update lib/pure/includes/osenv.nim

* fixing cstring

Co-authored-by: Andreas Rumpf <[email protected]>

fix nim-lang#19343 (#19344) [backport]

Ensure HttpClient onProgress is called once per second
Ensure that reported speed is accurate

stylecheck usages part two: stdlib cleanup (#19338)

* stylecheck usages part two: stdlib cleanup

typeinfo.nim: importCompilerProc => importcompilerproc

nre.nim: newLineFlags => newlineFlags

system.nim: JSRoot => JsRoot

ref #19319

* prefer importCompilerProc

added filemode docs (#19346)

Fix `remove` on last node of singly-linked list [backport:1.6] (#19353)

fix stylecheck error with asyncdispatch (#19350)

* stylecheck usages part two: stdlib cleanup

typeinfo.nim: importCompilerProc => importcompilerproc

nre.nim: newLineFlags => newlineFlags

system.nim: JSRoot => JsRoot

ref #19319

* prefer importCompilerProc

* fix stylecheck error with asyncdispatch

it is a partial regression since #12842

* add tests

* don't use echo in tests

remove spaces between an identifier and a star (#19355)

It makes search easier by searching `+`* instead of `+` which filter lots of unexported versions.

Follow nim-lang/Nim#18681

bitsets.nim: cleanup (#19361)

make rst thread safe (#19369)

split for the convenience of review

docs: Mention `import foo {.all.}` syntax (#19377)

Mention the `import foo {.all.}` syntax in the manual,
with a caveat about private imports.
Also link to the experimental importutils module.

Co-authored-by: adigitoleo <[email protected]>

update copyright year (#19381)

docs: Fix broken cross references to `rfind` in strutils (#19382) [backport]

Fixes three broken cross references to `rfind` in strutils.
Breakage due to signature changes of the `rfind` methods.

Co-authored-by: adigitoleo <[email protected]>

move type operation section and remove deepcopy document (#19389)

ref #19173; because deepcopy is not fit for ORC/ARC which was used for spawn and spawn will be removed from compiler

deprecate unsafeAddr; extend addr (#19373)

* deprecate unsafeAddr; extend addr

addr is now available for all addressable locations, unsafeAddr is deprecated and become an alias for addr

* follow @Vindaar's advice

* change the signature of addr

* unsafeAddr => addr (stdlib)

* Update changelog.md

* unsafeAddr => addr (tests)

* Revert "unsafeAddr => addr (stdlib)"

This reverts commit ab83c99c507048a8396e636bf22d55fdd84d7d1c.

* doc changes; thanks to @konsumlamm

Co-authored-by: konsumlamm <[email protected]>

Co-authored-by: konsumlamm <[email protected]>

improve changelog a bit (#19400)

mangle names in nimbase.h using cppDefine (#19395) [backport]

mangle names in nimbase.h
fix comments

Optimize lent in JS [backport:1.6] (#19393)

* Optimize lent in JS [backport:1.6]

* addr on lent doesn't work anymore, don't use it

* use unsafeAddr  in test again for older versions

update deprecated example (#19415)

`toNimIdent` proc is deprecated, so I replaced it with `ident` proc

Improve Zshell completion (#19354)

fix stricteffects (nimsuggest/sexp) (#19405)

* fix stricteffects (nimsuggest/sexp)

* Update tstrict_effects3.nim

* Update tests/effects/tstrict_effects3.nim

suppress deprecated warnings (#19408)

* suppress deprecated warnings

once bump version to 1.7.3 enable deprecated messages

* deprecate later

add an example to setControlCHook (#19416)

* add an example to setControlCHook

* [skip CI] format example for setControlCHook

Co-authored-by: Nathan Blaxall <[email protected]>

fix term rewriting with sideeffect (#19410)

* fix term rewriting with sideeffect

fix #6217

* add tests

* Update tests/template/template_various.nim

Resolve cross file resolution errors in atomics (#19422) [backport:1.6]

* Resolve call undeclared routine testAndSet

* Fix undeclared field atomicType

Fix #11923 (#19427)

* Apply commit nim-lang/Nim@5da931f that was never merged (was part of a bigger PR). Should fix issue #11932

* add a generic object for custom pragma

os: faster getFileSize (#19438)

Use "stat" rather than "open", "seek", and "close" system calls.
The Windows implementation remains the same.

bugfix: varargs count as open arrays (#19447)

update outdated link (#19465)

Ref nim-lang/Nim#19463

Update jsfetch with latest API and fix missing bindings (#19473)

* Update with latest API and fix missing bindings

remove deprecated `Body`
remove implicit `cstring` convs
add `Headers` to `FetchOptions`
add `Request` init proc which takes `FetchOptions`

* Update lib/std/jsfetch.nim

Co-authored-by: Juan Carlos <[email protected]>

* Update lib/std/jsfetch.nim

Co-authored-by: Juan Carlos <[email protected]>

* remove experimental flag

Co-authored-by: Juan Carlos <[email protected]>

No longer segfault when using a typeclass with a self referencing type (#19467)

Clonkk fix2 11923 (#19451)

* fix nnkBracketExpr not compiling for getImpl on customPragmaNode

* fix test import

* fix alias not working with hasCustomPragmas

fix parseEnum cannot parse enum with const fields (#19466)

fix #19463

Add compilers and hints to default nim.cfg (#18424)

don't use a temp for addr [backport: 1.6] (#19503)

* don't use a temp for addr

fix #19497

* Update compiler/ccgcalls.nim

Co-authored-by: konsumlamm <[email protected]>

* add a test

Co-authored-by: konsumlamm <[email protected]>

fixes #19404 by protecting the memory we borrow from. this replaces crashes with minor memory leaks which seems to be acceptable. In the longer run we need a better VM that didn't grow hacks over a decade. (#19515)

Co-authored-by: flywind <[email protected]>

Remove backslash in glob pattern (#19524)

use OrderedTable instead of OrderedTableRef for mimedb (#19522)

* use OrderedTable instead of OrderedTableRef for mimedb

Signed-off-by: David Krause <[email protected]>

* added changelog entry for mimedb change

Signed-off-by: David Krause <[email protected]>

Remove Deprecated oids.oidsToString (#19519)

* Remove deprecated oids.oidToString

* Remove deprecated oids.oidToString

Remove deprecated math.c_frexp (#19518)

* Remove Deprecated math proc

* Remove Deprecated math proc

* Remove Deprecated math proc

[testcase] genSym fails to make unique identifier for ref object types (#19506)

close #15118

Documentation: Fix word usage (#19529)

Update chcks.nim (#19540)

keep casing of noinit and noreturn pragmas consistently documented (#19535)

compile pragma: cache the result sooner (#19554)

extccomp.addExternalFileToCompile() relies on hashes to decide whether
an external C file needs recompilation or not.

Due to short-circuit evaluation of boolean expressions, the procedure
that generates a corresponding hash file is not called the first time an
external file is compiled, so an avoidable recompilation is triggered
the next build.

This patch fixes that by moving the proc call with a desired side
effect from its boolean expression, so it's executed unconditionally.
Clyybber pushed a commit to Clyybber/nimskull that referenced this pull request Feb 25, 2022
Concerns these changes:
nim-lang/Nim@727c637...340b5a1

Excluded changes are:
nim-lang/Nim#18963
nim-lang/Nim#19003
nim-lang/Nim#19043
nim-lang/Nim#19055
nim-lang/Nim#19053
nim-lang/Nim#19064
nim-lang/Nim#18642
nim-lang/Nim#19062
nim-lang/Nim#19082
nim-lang/Nim#19090
nim-lang/Nim#19077
nim-lang/Nim#19021
nim-lang/Nim#19100
nim-lang/Nim#19102
nim-lang/Nim#19111
nim-lang/Nim#19115
nim-lang/Nim#19133
nim-lang/Nim#19142
nim-lang/Nim#19158
nim-lang/Nim#19129
nim-lang/Nim#19137
nim-lang/Nim#19168
nim-lang/Nim#19156
nim-lang/Nim#19147
nim-lang/Nim#19180
nim-lang/Nim#19183
nim-lang/Nim#19182
nim-lang/Nim#19187
nim-lang/Nim#19179
nim-lang/Nim#19209
nim-lang/Nim#19210
nim-lang/Nim#19207
nim-lang/Nim#19219
nim-lang/Nim#19195
nim-lang/Nim#19212
nim-lang/Nim#19134
nim-lang/Nim#19235
nim-lang/Nim#19252
nim-lang/Nim#19196
nim-lang/Nim#19295
nim-lang/Nim#19301
nim-lang/Nim#19181
nim-lang/Nim#17223
nim-lang/Nim#19370
nim-lang/Nim#19385
nim-lang/Nim#19307
nim-lang/Nim#19394
nim-lang/Nim#19399
nim-lang/Nim#19390
nim-lang/Nim#19407
nim-lang/Nim#19419
nim-lang/Nim#19421
nim-lang/Nim#19363
nim-lang/Nim#19406
nim-lang/Nim#19431
nim-lang/Nim#19455
nim-lang/Nim#19461
nim-lang/Nim@cb894c7
nim-lang/Nim#19462
nim-lang/Nim#19442
nim-lang/Nim#19437
nim-lang/Nim#19433
nim-lang/Nim#19512
nim-lang/Nim#19487
nim-lang/Nim#19543

Excluded changes include major changes which require more consideration
and changes which don't apply to the current code anymore but could be
worth porting over still.

Excluded changes which only change the identifier casing in tests
or only concern code removed in nimskull aren't listed.

Begin commit listing:

use two underscores for easy demangling [backport:1.6] (#19028)

Add Elbrus 2000 architecture (#19024)

* Add Elbrus 2000 architecture

* Add e2k to niminst

* Update compiler/installer.ini

Co-authored-by: Andreas Rumpf <[email protected]>

remove exception (#18906)

allow converting static vars to `openArray` (#19035)

When assigning constant output to a seq, and then passing that static
seq to other functions that take `openArray`, the compiler may end up
producing errors, as it does not know how to convert `static[seq[T]]`
to `openArray[T]`. By ignoring the `static` wrapper on the type for
the purpose of determining data memory location and length, this gets
resolved cleanly. Unfortunately, it is relatively tricky to come up
with a minimal example, as there are followup problems from the failing
conversion, e.g., this may lead to `internal error: inconsistent
environment type`, instead of the relevant `openArrayLoc` error message.

use the correct header for TIOCGWINSZ on Solaris (#19037)

Minor update to terminal docs (#19056)

* Update terminal.nim

- Added some extra docs to cursorUp/Down/Forward/Backward
- I was able to use hideCursor and showCursor without adding stdout, removed the parameter
- Added docs to terminalHeight()* and terminalWidth()*

* Update lib/pure/terminal.nim

Co-authored-by: konsumlamm <[email protected]>

* Update lib/pure/terminal.nim

Co-authored-by: konsumlamm <[email protected]>

* Added back f: file to cursor movement

* Removed unnecessary comments

Co-authored-by: konsumlamm <[email protected]>

fix a tiny formating issue in doc/destructors.rst (#19058)

fix a tiny code snippet formatting issue in `doc/constructors.rst`, again (#19065)

Fix nimIdentNormalize, fixes #19067 (#19068)

* Make nimIdentNormalize return "" when passed ""; fixes #19067

Fixes #19067

* Add tests for nimIdentNormalize

fix #18971 (#19070) [backport:1.6]

since the example code return value from global variable, instead
of first argument, the `n.len` is 1 which causes compiler crashes.

fixes #19000 (#19032)

* fixes #19000

* progress

fix #18410 (Errors initializing an object of RootObj with the C++ backend) [backport] (#18836)

* fix #18410

* one line comment

* typo

* typo

* cover cpp

update numbers of lifetime-tracking hooks in doc/destructors.rst (#19088)

bootstrapping Nim compiler with `cpp --gc:orc` (#19087)

libs/impore/re: Add note about the requirement of `matches` to be pre-allocated (#19081)

Add few runnableExamples for `findBounds` for clarity.

Fixes nim-lang/Nim#18775

Add test for issue 15435 (#19079)

* Add test for issue 15435

Closes nim-lang/Nim#15435.

* Specify bug # in comment

Addresses nim-lang/Nim#19079 (comment)

manual: Document the use of `static` as a proc call (#19084)

* manual: Document the use of `static` as a proc call

Also adds tests.

Fixes nim-lang/Nim#16987 .

* Update doc/manual.rst

Co-authored-by: konsumlamm <[email protected]>

* Use the "bug #NNNN" comment syntax for consistency

Ref:
https://nim-lang.github.io/Nim/contributing.html#writing-tests-stdlib

> Always refer to a GitHub issue using the following exact syntax: bug
for tooling.

* manual: Undocument usage of foo.static

foo.static and foo.static() are not expected to work.

Ref: https://github.com/nim-lang/Nim/pull/19084/files#r741203578

Co-authored-by: konsumlamm <[email protected]>

manual: Document that comma propagates the default values of parameters (#19080)

* manual: Document that comma propagates the default values of parameters

Fixes nim-lang/Nim#15949.

* Use the "bug #NNNN" comment syntax for consistency

Ref:
https://nim-lang.github.io/Nim/contributing.html#writing-tests-stdlib

> Always refer to a GitHub issue using the following exact syntax: bug
for tooling.

fixes #19011 [backport:1.6] (#19114)

Add deprecation pragmas in lib/deprecated/pure (#19113)

Deprecate `std/sharedlist` and `std/sharedtables` (#19112)

fix nimindexterm in rst2tex/doc2tex [backport] (#19106)

* fix nimindexterm (rst2tex/doc2tex) [backport]

* Add support for indexing in rst

Call {.cursor.} a pragma. (#19116)

* Call {.cursor.} a pragma.

Its hard to find .curser annotation while googling because all other things like it are called pragmas. See https://nim-lang.org/docs/manual.html#pragmas
Also the . in front of the name makes it hard to find and search for.

Can we just call it cursor pragma?

* Small fix for comment.

Remove tlsEmulation enabled from Windows + GCC config (#19119) [backport:1.6]

This flag has a very significant performance impact on programs compiled with --threads:on. It is also apparently not needed anymore for standard circumstances. Can we remove the config? See nim-lang/Nim#18146 (comment) for discussion and perf impact. [backport:1.6]

Add security tip for setCookie (#19117)

* Add security tip for setCookie

* Update lib/pure/cookies.nim

Co-authored-by: Dominik Picheta <[email protected]>

* Update lib/pure/cookies.nim

Co-authored-by: konsumlamm <[email protected]>

Co-authored-by: Andreas Rumpf <[email protected]>
Co-authored-by: Dominik Picheta <[email protected]>
Co-authored-by: konsumlamm <[email protected]>

correct cookie docs (#19122)

refactoring: orc can use getThreadId() (#19123)

* refactoring: orc can use getThreadId()

* progress

fixed colorNames sorting mistake (#19125) [backport]

update manual (#19130) [backport]

Merge file size fields correctly on Windows (#19141)

* Merge file size fields correctly on Windows

Merge file size fields correctly on Windows

- Merge the two 32-bit file size fields from `BY_HANDLE_FILE_INFORMATION` correctly in `rawToFormalFileInfo`.
- Fixes #19135

* Update os.nim

Fix punycode.decode function (#19136)

* Refactor: rename proc to func

* Fix punycode.decode function

This function could only properly decode punycodes containing a single
encoded unicode character. As soon as there was more than one punycode
character group to decode it produced invalid output - the number of
characters was correct, but their position was not.

* Update tpunycode.nim

Co-authored-by: Clay Sweetser <[email protected]>

Fix undeclared 'SYS_getrandom' on emscripten (#19144)

wrong spaces (3 => 2) (#19145)

`caseStmtMacros` no longer experimental, experimental manual refactor (#19173)

* `caseStmtMacros` no longer experimental, experimental manual refactor

* Update doc/manual.rst

* apply review suggestions

* apply review

Co-authored-by: Andreas Rumpf <[email protected]>

fix inline syntax highlighting in system.nim (#19184)

swap port to correct port order (#19177)

Co-authored-by: Jaremy Creechley <[email protected]>

feat: TLS-ALPN wrappers for OpenSSL (#19202)

Co-authored-by: Iced Quinn <[email protected]>

misc bugfixes [backport:1.2] (#19203)

treat do with pragmas but no parens as proc (#19191)

fixes #19188

[format minor] remove unnecessary spaces (#19216)

Making TCC work again on Windows --cpu:amd64 - fix #16326 (#19221)

* fix #16326

* removing comments

fixes a converter handling regression that caused private converters to leak into client modules; fixes #19213; [backport:1.6] (#19229)

Add support for LoongArch (#19223)

* Add support for LoongArch

* Update compiler/installer.ini

Co-authored-by: Andreas Rumpf <[email protected]>

remove `std/sharedstrings` (#19228)

* remove std/sharedstrings

it has been broken since 0.18.0

* rephrase the changelog entry

add comments to spawn and pinnedSpawn (#19230)

`spawn` uses `nimSpawn3` internally and `pinnedSpawn` uses `nimSpawn4` internally. I comment it in order to help contributors get the gist of its functionality.

fixes an old ARC bug: the produced copy/sink operations don't copy the hidden type field for objects with enabled inheritance; fixes #19205 [backport:1.6] (#19232)

nimRawSetjmp: support Windows (#19197)

* nimRawSetjmp: support Windows

Using `_setjmp()` directly is required to avoid some rare (but very
annoying) exception-related stack corruption leading to segfaults on
Windows, with Mingw-w64 and SEH.
More details: status-im/nimbus-eth2#3121

Also add "nimBuiltinSetjmp" - mostly for benchmarking.

* fix for Apple's Clang++

Revert "swap port to correct port order (#19177)" (#19234)

This reverts commit 0d0c249.

move toDeque to after addLast (#19233) [backport:1.0]

Changes the order of procs definitions in order to avoid calling an undefined proc.

let Nim support Nimble 0.14 with lock-file support [backport:1.6] (#19236)

nimc.rst: fix table markup (#19239)

Various std net improvements (#19132)

* Variant of  that works with raw IpAddresses.

- Add doc tests for new net proc's.
- Aadd recvFrom impl
- Add recvFrom impl -- tweak handling data var

- Update lib/pure/net.nim
	Co-authored-by: Dominik Picheta <[email protected]>

- cleaning up sendTo args
- remove extra connect test
- cleaning up sendTo args
- fix inet_ntop test
- fix test failing - byte len

* fix test failing - byte len

* debugging odd windows build failure

* debugging odd windows build failure

* more experiments to figure out the windows failure

* try manual assigment on InAddr

Co-authored-by: Jaremy Creechley <[email protected]>

fix bug #14468 zero-width split (#19248)

basicopt.txt: Unify the format (#19251)

fix: fixes bug in CVerifyPeerUseEnvVars (#19247)

Previously CVerifyPeerUseEnvVars was not being passed into
scanSslCertificates, which meant that we weren't scanning
additional certificate locations given via the SSL_CERT_FILE and
SSL_CERT_DIR environment variables

suggestion to respect typedarray type (#19257)

* suggestion to respect typedarray

* Update jssys.nim

Co-authored-by: Sven Keller <[email protected]>

fix #19244 - solves the problem of the InAddr object constructor in Windows. (#19259)

* Update winlean.nim

* Update tnet_ll.nim

Fixed typo in manual.rst unsafeAssign->uncheckedAssign. Fixes part 1 of #19266 (#19267)

use uppercase "type" for Proxy-Authorization header (#19273)

Some servers will reject authorization requests with a lowercase "basic" type. Changing to "Basic" seems to solve these issues.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization

Update colors.nim (#19274)

* Update colors.nim

Added `lightgray` alias to `lightgrey` and `...grey`aliases for the rest of the gray colors.
Added color `rebeccapurple`.
Fixed the incorrect values for the `PaleVioletRed` and `MediumPurple` colors.
This module should now be matching the CSS colors.
I used the seq[tuple] syntax for defining the names.

* Document colors changes.

Extract runnables that specify `doccmd` (#19275) [backport:1.6]

Fix build on FreeBSD/powerpc (#19282)

It's currently misdetected as powerpc64.

Fix #19107 (#19286) [backport]

fixes grammar typos [backport] (#19289)

fix 19292 (#19293)

Fix #19297 - fixing broken list after adding empty list (#19299)

* Update lists.nim

* Update tlists.nim

* removed check `if b.tail != nil`

The tail of the list being null it is still possible to retrieve its end by going through all nodes from the head. So checking for null from `b.tail` is unnecessary. However, setting `a.tail = b.tail` only if `a.head != nil`, so you don't break a good list with an already broken one.

fixes #16617 [backport] (#19300)

Update JS and nimscript import tests (#19306)

* add new modules, except experimental ones
* remove deprecated modules mersenne and sharedlist
* better describe why some modules fail and some modules don't

add compile time option for POSIX sigwait on Illumos/Solaris (#19296)

* add compile time option for POSIX sigwait on Illumos/Solaris

* fix link to documentation of `sigwait` on Illumos/Solaris

[docs] clarify the raised exception (#19308)

* [docs] clarify the raised exception

Lest developers wanna know what the exception is.

* Apply suggestions from @konsumlamm

Co-authored-by: konsumlamm <[email protected]>

Co-authored-by: konsumlamm <[email protected]>

docs: Fix typo in tut1.rst (#19309)

Fix #19314 - fixing broken `DoublyLinkedList` after adding empty `DoublyLinkedList` (#19315) [backport]

* Update lists.nim

* Update tlists.nim

fixed typos (#19316)

devel: style fix (#19318)

this allows "--styleCheck:usages --styleCheck:error"

docs: Fix typo in tut1.rst (#19324)

correct the comments (#19322)

--expandArc

```
var
  a
  b
a = matrix(5, 5, 1.0)
b = matrix(5, 5, 2.0)
`=sink`(b, -
  let blitTmp = b
  wasMoved(b)
  blitTmp +
    a)
`=destroy`(b)
`=destroy`(a)
```

add std/private/win_getsysteminfo; refactor the usage of `GetSystemInfo` (#19310)

* add std/private/win_getsysteminfo

* import at the top level

* wrappers follow nep1 too

* follow review comment

Update net.nim (#19327) [backport]

Fix #19038 - making the Nim compiler work again on Windows XP (#19331)

* Update osenv.nim

* Update win_setenv.nim

* Update lib/pure/includes/osenv.nim

* Update lib/pure/includes/osenv.nim

* fixing cstring

Co-authored-by: Andreas Rumpf <[email protected]>

fix nim-lang#19343 (#19344) [backport]

Ensure HttpClient onProgress is called once per second
Ensure that reported speed is accurate

stylecheck usages part two: stdlib cleanup (#19338)

* stylecheck usages part two: stdlib cleanup

typeinfo.nim: importCompilerProc => importcompilerproc

nre.nim: newLineFlags => newlineFlags

system.nim: JSRoot => JsRoot

ref #19319

* prefer importCompilerProc

added filemode docs (#19346)

Fix `remove` on last node of singly-linked list [backport:1.6] (#19353)

fix stylecheck error with asyncdispatch (#19350)

* stylecheck usages part two: stdlib cleanup

typeinfo.nim: importCompilerProc => importcompilerproc

nre.nim: newLineFlags => newlineFlags

system.nim: JSRoot => JsRoot

ref #19319

* prefer importCompilerProc

* fix stylecheck error with asyncdispatch

it is a partial regression since #12842

* add tests

* don't use echo in tests

remove spaces between an identifier and a star (#19355)

It makes search easier by searching `+`* instead of `+` which filter lots of unexported versions.

Follow nim-lang/Nim#18681

bitsets.nim: cleanup (#19361)

make rst thread safe (#19369)

split for the convenience of review

docs: Mention `import foo {.all.}` syntax (#19377)

Mention the `import foo {.all.}` syntax in the manual,
with a caveat about private imports.
Also link to the experimental importutils module.

Co-authored-by: adigitoleo <[email protected]>

update copyright year (#19381)

docs: Fix broken cross references to `rfind` in strutils (#19382) [backport]

Fixes three broken cross references to `rfind` in strutils.
Breakage due to signature changes of the `rfind` methods.

Co-authored-by: adigitoleo <[email protected]>

move type operation section and remove deepcopy document (#19389)

ref #19173; because deepcopy is not fit for ORC/ARC which was used for spawn and spawn will be removed from compiler

deprecate unsafeAddr; extend addr (#19373)

* deprecate unsafeAddr; extend addr

addr is now available for all addressable locations, unsafeAddr is deprecated and become an alias for addr

* follow @Vindaar's advice

* change the signature of addr

* unsafeAddr => addr (stdlib)

* Update changelog.md

* unsafeAddr => addr (tests)

* Revert "unsafeAddr => addr (stdlib)"

This reverts commit ab83c99c507048a8396e636bf22d55fdd84d7d1c.

* doc changes; thanks to @konsumlamm

Co-authored-by: konsumlamm <[email protected]>

Co-authored-by: konsumlamm <[email protected]>

improve changelog a bit (#19400)

mangle names in nimbase.h using cppDefine (#19395) [backport]

mangle names in nimbase.h
fix comments

Optimize lent in JS [backport:1.6] (#19393)

* Optimize lent in JS [backport:1.6]

* addr on lent doesn't work anymore, don't use it

* use unsafeAddr  in test again for older versions

update deprecated example (#19415)

`toNimIdent` proc is deprecated, so I replaced it with `ident` proc

Improve Zshell completion (#19354)

fix stricteffects (nimsuggest/sexp) (#19405)

* fix stricteffects (nimsuggest/sexp)

* Update tstrict_effects3.nim

* Update tests/effects/tstrict_effects3.nim

suppress deprecated warnings (#19408)

* suppress deprecated warnings

once bump version to 1.7.3 enable deprecated messages

* deprecate later

add an example to setControlCHook (#19416)

* add an example to setControlCHook

* [skip CI] format example for setControlCHook

Co-authored-by: Nathan Blaxall <[email protected]>

fix term rewriting with sideeffect (#19410)

* fix term rewriting with sideeffect

fix #6217

* add tests

* Update tests/template/template_various.nim

Resolve cross file resolution errors in atomics (#19422) [backport:1.6]

* Resolve call undeclared routine testAndSet

* Fix undeclared field atomicType

Fix #11923 (#19427)

* Apply commit nim-lang/Nim@5da931f that was never merged (was part of a bigger PR). Should fix issue #11932

* add a generic object for custom pragma

os: faster getFileSize (#19438)

Use "stat" rather than "open", "seek", and "close" system calls.
The Windows implementation remains the same.

bugfix: varargs count as open arrays (#19447)

update outdated link (#19465)

Ref nim-lang/Nim#19463

Update jsfetch with latest API and fix missing bindings (#19473)

* Update with latest API and fix missing bindings

remove deprecated `Body`
remove implicit `cstring` convs
add `Headers` to `FetchOptions`
add `Request` init proc which takes `FetchOptions`

* Update lib/std/jsfetch.nim

Co-authored-by: Juan Carlos <[email protected]>

* Update lib/std/jsfetch.nim

Co-authored-by: Juan Carlos <[email protected]>

* remove experimental flag

Co-authored-by: Juan Carlos <[email protected]>

No longer segfault when using a typeclass with a self referencing type (#19467)

Clonkk fix2 11923 (#19451)

* fix nnkBracketExpr not compiling for getImpl on customPragmaNode

* fix test import

* fix alias not working with hasCustomPragmas

fix parseEnum cannot parse enum with const fields (#19466)

fix #19463

Add compilers and hints to default nim.cfg (#18424)

don't use a temp for addr [backport: 1.6] (#19503)

* don't use a temp for addr

fix #19497

* Update compiler/ccgcalls.nim

Co-authored-by: konsumlamm <[email protected]>

* add a test

Co-authored-by: konsumlamm <[email protected]>

fixes #19404 by protecting the memory we borrow from. this replaces crashes with minor memory leaks which seems to be acceptable. In the longer run we need a better VM that didn't grow hacks over a decade. (#19515)

Co-authored-by: flywind <[email protected]>

Remove backslash in glob pattern (#19524)

use OrderedTable instead of OrderedTableRef for mimedb (#19522)

* use OrderedTable instead of OrderedTableRef for mimedb

Signed-off-by: David Krause <[email protected]>

* added changelog entry for mimedb change

Signed-off-by: David Krause <[email protected]>

Remove Deprecated oids.oidsToString (#19519)

* Remove deprecated oids.oidToString

* Remove deprecated oids.oidToString

Remove deprecated math.c_frexp (#19518)

* Remove Deprecated math proc

* Remove Deprecated math proc

* Remove Deprecated math proc

[testcase] genSym fails to make unique identifier for ref object types (#19506)

close #15118

Documentation: Fix word usage (#19529)

Update chcks.nim (#19540)

keep casing of noinit and noreturn pragmas consistently documented (#19535)

compile pragma: cache the result sooner (#19554)

extccomp.addExternalFileToCompile() relies on hashes to decide whether
an external C file needs recompilation or not.

Due to short-circuit evaluation of boolean expressions, the procedure
that generates a corresponding hash file is not called the first time an
external file is compiled, so an avoidable recompilation is triggered
the next build.

This patch fixes that by moving the proc call with a desired side
effect from its boolean expression, so it's executed unconditionally.
Clyybber pushed a commit to Clyybber/nimskull that referenced this pull request Feb 25, 2022
Concerns these changes:
nim-lang/Nim@727c637...340b5a1

Excluded changes are:
nim-lang/Nim#18963
nim-lang/Nim#19003
nim-lang/Nim#19043
nim-lang/Nim#19055
nim-lang/Nim#19053
nim-lang/Nim#19064
nim-lang/Nim#18642
nim-lang/Nim#19062
nim-lang/Nim#19082
nim-lang/Nim#19090
nim-lang/Nim#19077
nim-lang/Nim#19021
nim-lang/Nim#19100
nim-lang/Nim#19102
nim-lang/Nim#19111
nim-lang/Nim#19115
nim-lang/Nim#19133
nim-lang/Nim#19142
nim-lang/Nim#19158
nim-lang/Nim#19129
nim-lang/Nim#19137
nim-lang/Nim#19168
nim-lang/Nim#19156
nim-lang/Nim#19147
nim-lang/Nim#19180
nim-lang/Nim#19183
nim-lang/Nim#19182
nim-lang/Nim#19187
nim-lang/Nim#19179
nim-lang/Nim#19209
nim-lang/Nim#19210
nim-lang/Nim#19207
nim-lang/Nim#19219
nim-lang/Nim#19195
nim-lang/Nim#19212
nim-lang/Nim#19134
nim-lang/Nim#19235
nim-lang/Nim#19252
nim-lang/Nim#19196
nim-lang/Nim#19295
nim-lang/Nim#19301
nim-lang/Nim#19181
nim-lang/Nim#17223
nim-lang/Nim#19370
nim-lang/Nim#19385
nim-lang/Nim#19307
nim-lang/Nim#19394
nim-lang/Nim#19399
nim-lang/Nim#19390
nim-lang/Nim#19407
nim-lang/Nim#19419
nim-lang/Nim#19421
nim-lang/Nim#19363
nim-lang/Nim#19406
nim-lang/Nim#19431
nim-lang/Nim#19455
nim-lang/Nim#19461
nim-lang/Nim@cb894c7
nim-lang/Nim#19462
nim-lang/Nim#19442
nim-lang/Nim#19437
nim-lang/Nim#19433
nim-lang/Nim#19512
nim-lang/Nim#19487
nim-lang/Nim#19543

Excluded changes include major changes which require more consideration
and changes which don't apply to the current code anymore but could be
worth porting over still.

Excluded changes which only change the identifier casing in tests
or only concern code removed in nimskull aren't listed.

Begin commit listing:

use two underscores for easy demangling [backport:1.6] (#19028)

Add Elbrus 2000 architecture (#19024)

* Add Elbrus 2000 architecture

* Add e2k to niminst

* Update compiler/installer.ini

Co-authored-by: Andreas Rumpf <[email protected]>

remove exception (#18906)

allow converting static vars to `openArray` (#19035)

When assigning constant output to a seq, and then passing that static
seq to other functions that take `openArray`, the compiler may end up
producing errors, as it does not know how to convert `static[seq[T]]`
to `openArray[T]`. By ignoring the `static` wrapper on the type for
the purpose of determining data memory location and length, this gets
resolved cleanly. Unfortunately, it is relatively tricky to come up
with a minimal example, as there are followup problems from the failing
conversion, e.g., this may lead to `internal error: inconsistent
environment type`, instead of the relevant `openArrayLoc` error message.

use the correct header for TIOCGWINSZ on Solaris (#19037)

Minor update to terminal docs (#19056)

* Update terminal.nim

- Added some extra docs to cursorUp/Down/Forward/Backward
- I was able to use hideCursor and showCursor without adding stdout, removed the parameter
- Added docs to terminalHeight()* and terminalWidth()*

* Update lib/pure/terminal.nim

Co-authored-by: konsumlamm <[email protected]>

* Update lib/pure/terminal.nim

Co-authored-by: konsumlamm <[email protected]>

* Added back f: file to cursor movement

* Removed unnecessary comments

Co-authored-by: konsumlamm <[email protected]>

fix a tiny formating issue in doc/destructors.rst (#19058)

fix a tiny code snippet formatting issue in `doc/constructors.rst`, again (#19065)

Fix nimIdentNormalize, fixes #19067 (#19068)

* Make nimIdentNormalize return "" when passed ""; fixes #19067

Fixes #19067

* Add tests for nimIdentNormalize

fix #18971 (#19070) [backport:1.6]

since the example code return value from global variable, instead
of first argument, the `n.len` is 1 which causes compiler crashes.

fixes #19000 (#19032)

* fixes #19000

* progress

fix #18410 (Errors initializing an object of RootObj with the C++ backend) [backport] (#18836)

* fix #18410

* one line comment

* typo

* typo

* cover cpp

update numbers of lifetime-tracking hooks in doc/destructors.rst (#19088)

bootstrapping Nim compiler with `cpp --gc:orc` (#19087)

libs/impore/re: Add note about the requirement of `matches` to be pre-allocated (#19081)

Add few runnableExamples for `findBounds` for clarity.

Fixes nim-lang/Nim#18775

Add test for issue 15435 (#19079)

* Add test for issue 15435

Closes nim-lang/Nim#15435.

* Specify bug # in comment

Addresses nim-lang/Nim#19079 (comment)

manual: Document the use of `static` as a proc call (#19084)

* manual: Document the use of `static` as a proc call

Also adds tests.

Fixes nim-lang/Nim#16987 .

* Update doc/manual.rst

Co-authored-by: konsumlamm <[email protected]>

* Use the "bug #NNNN" comment syntax for consistency

Ref:
https://nim-lang.github.io/Nim/contributing.html#writing-tests-stdlib

> Always refer to a GitHub issue using the following exact syntax: bug
for tooling.

* manual: Undocument usage of foo.static

foo.static and foo.static() are not expected to work.

Ref: https://github.com/nim-lang/Nim/pull/19084/files#r741203578

Co-authored-by: konsumlamm <[email protected]>

manual: Document that comma propagates the default values of parameters (#19080)

* manual: Document that comma propagates the default values of parameters

Fixes nim-lang/Nim#15949.

* Use the "bug #NNNN" comment syntax for consistency

Ref:
https://nim-lang.github.io/Nim/contributing.html#writing-tests-stdlib

> Always refer to a GitHub issue using the following exact syntax: bug
for tooling.

fixes #19011 [backport:1.6] (#19114)

Add deprecation pragmas in lib/deprecated/pure (#19113)

Deprecate `std/sharedlist` and `std/sharedtables` (#19112)

fix nimindexterm in rst2tex/doc2tex [backport] (#19106)

* fix nimindexterm (rst2tex/doc2tex) [backport]

* Add support for indexing in rst

Call {.cursor.} a pragma. (#19116)

* Call {.cursor.} a pragma.

Its hard to find .curser annotation while googling because all other things like it are called pragmas. See https://nim-lang.org/docs/manual.html#pragmas
Also the . in front of the name makes it hard to find and search for.

Can we just call it cursor pragma?

* Small fix for comment.

Remove tlsEmulation enabled from Windows + GCC config (#19119) [backport:1.6]

This flag has a very significant performance impact on programs compiled with --threads:on. It is also apparently not needed anymore for standard circumstances. Can we remove the config? See nim-lang/Nim#18146 (comment) for discussion and perf impact. [backport:1.6]

Add security tip for setCookie (#19117)

* Add security tip for setCookie

* Update lib/pure/cookies.nim

Co-authored-by: Dominik Picheta <[email protected]>

* Update lib/pure/cookies.nim

Co-authored-by: konsumlamm <[email protected]>

Co-authored-by: Andreas Rumpf <[email protected]>
Co-authored-by: Dominik Picheta <[email protected]>
Co-authored-by: konsumlamm <[email protected]>

correct cookie docs (#19122)

refactoring: orc can use getThreadId() (#19123)

* refactoring: orc can use getThreadId()

* progress

fixed colorNames sorting mistake (#19125) [backport]

update manual (#19130) [backport]

Merge file size fields correctly on Windows (#19141)

* Merge file size fields correctly on Windows

Merge file size fields correctly on Windows

- Merge the two 32-bit file size fields from `BY_HANDLE_FILE_INFORMATION` correctly in `rawToFormalFileInfo`.
- Fixes #19135

* Update os.nim

Fix punycode.decode function (#19136)

* Refactor: rename proc to func

* Fix punycode.decode function

This function could only properly decode punycodes containing a single
encoded unicode character. As soon as there was more than one punycode
character group to decode it produced invalid output - the number of
characters was correct, but their position was not.

* Update tpunycode.nim

Co-authored-by: Clay Sweetser <[email protected]>

Fix undeclared 'SYS_getrandom' on emscripten (#19144)

wrong spaces (3 => 2) (#19145)

`caseStmtMacros` no longer experimental, experimental manual refactor (#19173)

* `caseStmtMacros` no longer experimental, experimental manual refactor

* Update doc/manual.rst

* apply review suggestions

* apply review

Co-authored-by: Andreas Rumpf <[email protected]>

fix inline syntax highlighting in system.nim (#19184)

swap port to correct port order (#19177)

Co-authored-by: Jaremy Creechley <[email protected]>

feat: TLS-ALPN wrappers for OpenSSL (#19202)

Co-authored-by: Iced Quinn <[email protected]>

misc bugfixes [backport:1.2] (#19203)

treat do with pragmas but no parens as proc (#19191)

fixes #19188

[format minor] remove unnecessary spaces (#19216)

Making TCC work again on Windows --cpu:amd64 - fix #16326 (#19221)

* fix #16326

* removing comments

fixes a converter handling regression that caused private converters to leak into client modules; fixes #19213; [backport:1.6] (#19229)

Add support for LoongArch (#19223)

* Add support for LoongArch

* Update compiler/installer.ini

Co-authored-by: Andreas Rumpf <[email protected]>

remove `std/sharedstrings` (#19228)

* remove std/sharedstrings

it has been broken since 0.18.0

* rephrase the changelog entry

add comments to spawn and pinnedSpawn (#19230)

`spawn` uses `nimSpawn3` internally and `pinnedSpawn` uses `nimSpawn4` internally. I comment it in order to help contributors get the gist of its functionality.

fixes an old ARC bug: the produced copy/sink operations don't copy the hidden type field for objects with enabled inheritance; fixes #19205 [backport:1.6] (#19232)

nimRawSetjmp: support Windows (#19197)

* nimRawSetjmp: support Windows

Using `_setjmp()` directly is required to avoid some rare (but very
annoying) exception-related stack corruption leading to segfaults on
Windows, with Mingw-w64 and SEH.
More details: status-im/nimbus-eth2#3121

Also add "nimBuiltinSetjmp" - mostly for benchmarking.

* fix for Apple's Clang++

Revert "swap port to correct port order (#19177)" (#19234)

This reverts commit 0d0c249.

move toDeque to after addLast (#19233) [backport:1.0]

Changes the order of procs definitions in order to avoid calling an undefined proc.

let Nim support Nimble 0.14 with lock-file support [backport:1.6] (#19236)

nimc.rst: fix table markup (#19239)

Various std net improvements (#19132)

* Variant of  that works with raw IpAddresses.

- Add doc tests for new net proc's.
- Aadd recvFrom impl
- Add recvFrom impl -- tweak handling data var

- Update lib/pure/net.nim
	Co-authored-by: Dominik Picheta <[email protected]>

- cleaning up sendTo args
- remove extra connect test
- cleaning up sendTo args
- fix inet_ntop test
- fix test failing - byte len

* fix test failing - byte len

* debugging odd windows build failure

* debugging odd windows build failure

* more experiments to figure out the windows failure

* try manual assigment on InAddr

Co-authored-by: Jaremy Creechley <[email protected]>

fix bug #14468 zero-width split (#19248)

basicopt.txt: Unify the format (#19251)

fix: fixes bug in CVerifyPeerUseEnvVars (#19247)

Previously CVerifyPeerUseEnvVars was not being passed into
scanSslCertificates, which meant that we weren't scanning
additional certificate locations given via the SSL_CERT_FILE and
SSL_CERT_DIR environment variables

suggestion to respect typedarray type (#19257)

* suggestion to respect typedarray

* Update jssys.nim

Co-authored-by: Sven Keller <[email protected]>

fix #19244 - solves the problem of the InAddr object constructor in Windows. (#19259)

* Update winlean.nim

* Update tnet_ll.nim

Fixed typo in manual.rst unsafeAssign->uncheckedAssign. Fixes part 1 of #19266 (#19267)

use uppercase "type" for Proxy-Authorization header (#19273)

Some servers will reject authorization requests with a lowercase "basic" type. Changing to "Basic" seems to solve these issues.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization

Update colors.nim (#19274)

* Update colors.nim

Added `lightgray` alias to `lightgrey` and `...grey`aliases for the rest of the gray colors.
Added color `rebeccapurple`.
Fixed the incorrect values for the `PaleVioletRed` and `MediumPurple` colors.
This module should now be matching the CSS colors.
I used the seq[tuple] syntax for defining the names.

* Document colors changes.

Extract runnables that specify `doccmd` (#19275) [backport:1.6]

Fix build on FreeBSD/powerpc (#19282)

It's currently misdetected as powerpc64.

Fix #19107 (#19286) [backport]

fixes grammar typos [backport] (#19289)

fix 19292 (#19293)

Fix #19297 - fixing broken list after adding empty list (#19299)

* Update lists.nim

* Update tlists.nim

* removed check `if b.tail != nil`

The tail of the list being null it is still possible to retrieve its end by going through all nodes from the head. So checking for null from `b.tail` is unnecessary. However, setting `a.tail = b.tail` only if `a.head != nil`, so you don't break a good list with an already broken one.

fixes #16617 [backport] (#19300)

Update JS and nimscript import tests (#19306)

* add new modules, except experimental ones
* remove deprecated modules mersenne and sharedlist
* better describe why some modules fail and some modules don't

add compile time option for POSIX sigwait on Illumos/Solaris (#19296)

* add compile time option for POSIX sigwait on Illumos/Solaris

* fix link to documentation of `sigwait` on Illumos/Solaris

[docs] clarify the raised exception (#19308)

* [docs] clarify the raised exception

Lest developers wanna know what the exception is.

* Apply suggestions from @konsumlamm

Co-authored-by: konsumlamm <[email protected]>

Co-authored-by: konsumlamm <[email protected]>

docs: Fix typo in tut1.rst (#19309)

Fix #19314 - fixing broken `DoublyLinkedList` after adding empty `DoublyLinkedList` (#19315) [backport]

* Update lists.nim

* Update tlists.nim

fixed typos (#19316)

devel: style fix (#19318)

this allows "--styleCheck:usages --styleCheck:error"

docs: Fix typo in tut1.rst (#19324)

correct the comments (#19322)

--expandArc

```
var
  a
  b
a = matrix(5, 5, 1.0)
b = matrix(5, 5, 2.0)
`=sink`(b, -
  let blitTmp = b
  wasMoved(b)
  blitTmp +
    a)
`=destroy`(b)
`=destroy`(a)
```

add std/private/win_getsysteminfo; refactor the usage of `GetSystemInfo` (#19310)

* add std/private/win_getsysteminfo

* import at the top level

* wrappers follow nep1 too

* follow review comment

Update net.nim (#19327) [backport]

Fix #19038 - making the Nim compiler work again on Windows XP (#19331)

* Update osenv.nim

* Update win_setenv.nim

* Update lib/pure/includes/osenv.nim

* Update lib/pure/includes/osenv.nim

* fixing cstring

Co-authored-by: Andreas Rumpf <[email protected]>

fix nim-lang#19343 (#19344) [backport]

Ensure HttpClient onProgress is called once per second
Ensure that reported speed is accurate

stylecheck usages part two: stdlib cleanup (#19338)

* stylecheck usages part two: stdlib cleanup

typeinfo.nim: importCompilerProc => importcompilerproc

nre.nim: newLineFlags => newlineFlags

system.nim: JSRoot => JsRoot

ref #19319

* prefer importCompilerProc

added filemode docs (#19346)

Fix `remove` on last node of singly-linked list [backport:1.6] (#19353)

fix stylecheck error with asyncdispatch (#19350)

* stylecheck usages part two: stdlib cleanup

typeinfo.nim: importCompilerProc => importcompilerproc

nre.nim: newLineFlags => newlineFlags

system.nim: JSRoot => JsRoot

ref #19319

* prefer importCompilerProc

* fix stylecheck error with asyncdispatch

it is a partial regression since #12842

* add tests

* don't use echo in tests

remove spaces between an identifier and a star (#19355)

It makes search easier by searching `+`* instead of `+` which filter lots of unexported versions.

Follow nim-lang/Nim#18681

bitsets.nim: cleanup (#19361)

make rst thread safe (#19369)

split for the convenience of review

docs: Mention `import foo {.all.}` syntax (#19377)

Mention the `import foo {.all.}` syntax in the manual,
with a caveat about private imports.
Also link to the experimental importutils module.

Co-authored-by: adigitoleo <[email protected]>

update copyright year (#19381)

docs: Fix broken cross references to `rfind` in strutils (#19382) [backport]

Fixes three broken cross references to `rfind` in strutils.
Breakage due to signature changes of the `rfind` methods.

Co-authored-by: adigitoleo <[email protected]>

move type operation section and remove deepcopy document (#19389)

ref #19173; because deepcopy is not fit for ORC/ARC which was used for spawn and spawn will be removed from compiler

deprecate unsafeAddr; extend addr (#19373)

* deprecate unsafeAddr; extend addr

addr is now available for all addressable locations, unsafeAddr is deprecated and become an alias for addr

* follow @Vindaar's advice

* change the signature of addr

* unsafeAddr => addr (stdlib)

* Update changelog.md

* unsafeAddr => addr (tests)

* Revert "unsafeAddr => addr (stdlib)"

This reverts commit ab83c99c507048a8396e636bf22d55fdd84d7d1c.

* doc changes; thanks to @konsumlamm

Co-authored-by: konsumlamm <[email protected]>

Co-authored-by: konsumlamm <[email protected]>

improve changelog a bit (#19400)

mangle names in nimbase.h using cppDefine (#19395) [backport]

mangle names in nimbase.h
fix comments

Optimize lent in JS [backport:1.6] (#19393)

* Optimize lent in JS [backport:1.6]

* addr on lent doesn't work anymore, don't use it

* use unsafeAddr  in test again for older versions

update deprecated example (#19415)

`toNimIdent` proc is deprecated, so I replaced it with `ident` proc

Improve Zshell completion (#19354)

fix stricteffects (nimsuggest/sexp) (#19405)

* fix stricteffects (nimsuggest/sexp)

* Update tstrict_effects3.nim

* Update tests/effects/tstrict_effects3.nim

suppress deprecated warnings (#19408)

* suppress deprecated warnings

once bump version to 1.7.3 enable deprecated messages

* deprecate later

add an example to setControlCHook (#19416)

* add an example to setControlCHook

* [skip CI] format example for setControlCHook

Co-authored-by: Nathan Blaxall <[email protected]>

fix term rewriting with sideeffect (#19410)

* fix term rewriting with sideeffect

fix #6217

* add tests

* Update tests/template/template_various.nim

Resolve cross file resolution errors in atomics (#19422) [backport:1.6]

* Resolve call undeclared routine testAndSet

* Fix undeclared field atomicType

Fix #11923 (#19427)

* Apply commit nim-lang/Nim@5da931f that was never merged (was part of a bigger PR). Should fix issue #11932

* add a generic object for custom pragma

os: faster getFileSize (#19438)

Use "stat" rather than "open", "seek", and "close" system calls.
The Windows implementation remains the same.

bugfix: varargs count as open arrays (#19447)

update outdated link (#19465)

Ref nim-lang/Nim#19463

Update jsfetch with latest API and fix missing bindings (#19473)

* Update with latest API and fix missing bindings

remove deprecated `Body`
remove implicit `cstring` convs
add `Headers` to `FetchOptions`
add `Request` init proc which takes `FetchOptions`

* Update lib/std/jsfetch.nim

Co-authored-by: Juan Carlos <[email protected]>

* Update lib/std/jsfetch.nim

Co-authored-by: Juan Carlos <[email protected]>

* remove experimental flag

Co-authored-by: Juan Carlos <[email protected]>

No longer segfault when using a typeclass with a self referencing type (#19467)

Clonkk fix2 11923 (#19451)

* fix nnkBracketExpr not compiling for getImpl on customPragmaNode

* fix test import

* fix alias not working with hasCustomPragmas

fix parseEnum cannot parse enum with const fields (#19466)

fix #19463

Add compilers and hints to default nim.cfg (#18424)

don't use a temp for addr [backport: 1.6] (#19503)

* don't use a temp for addr

fix #19497

* Update compiler/ccgcalls.nim

Co-authored-by: konsumlamm <[email protected]>

* add a test

Co-authored-by: konsumlamm <[email protected]>

fixes #19404 by protecting the memory we borrow from. this replaces crashes with minor memory leaks which seems to be acceptable. In the longer run we need a better VM that didn't grow hacks over a decade. (#19515)

Co-authored-by: flywind <[email protected]>

Remove backslash in glob pattern (#19524)

use OrderedTable instead of OrderedTableRef for mimedb (#19522)

* use OrderedTable instead of OrderedTableRef for mimedb

Signed-off-by: David Krause <[email protected]>

* added changelog entry for mimedb change

Signed-off-by: David Krause <[email protected]>

Remove Deprecated oids.oidsToString (#19519)

* Remove deprecated oids.oidToString

* Remove deprecated oids.oidToString

Remove deprecated math.c_frexp (#19518)

* Remove Deprecated math proc

* Remove Deprecated math proc

* Remove Deprecated math proc

[testcase] genSym fails to make unique identifier for ref object types (#19506)

close #15118

Documentation: Fix word usage (#19529)

Update chcks.nim (#19540)

keep casing of noinit and noreturn pragmas consistently documented (#19535)

compile pragma: cache the result sooner (#19554)

extccomp.addExternalFileToCompile() relies on hashes to decide whether
an external C file needs recompilation or not.

Due to short-circuit evaluation of boolean expressions, the procedure
that generates a corresponding hash file is not called the first time an
external file is compiled, so an avoidable recompilation is triggered
the next build.

This patch fixes that by moving the proc call with a desired side
effect from its boolean expression, so it's executed unconditionally.
bors bot added a commit to nim-works/nimskull that referenced this pull request Feb 26, 2022
240: Merge upstream changes r=Clyybber a=Clyybber

Concerns these changes:
nim-lang/Nim@727c637...340b5a1

Excluded changes are:
nim-lang/Nim#18963
nim-lang/Nim#19003
nim-lang/Nim#19043
nim-lang/Nim#19055
nim-lang/Nim#19053
nim-lang/Nim#19064
nim-lang/Nim#18642
nim-lang/Nim#19062
nim-lang/Nim#19082
nim-lang/Nim#19090
nim-lang/Nim#19077
nim-lang/Nim#19021
nim-lang/Nim#19100
nim-lang/Nim#19102
nim-lang/Nim#19111
nim-lang/Nim#19115
nim-lang/Nim#19133
nim-lang/Nim#19142
nim-lang/Nim#19158
nim-lang/Nim#19129
nim-lang/Nim#19137
nim-lang/Nim#19168
nim-lang/Nim#19156
nim-lang/Nim#19147
nim-lang/Nim#19180
nim-lang/Nim#19183
nim-lang/Nim#19182
nim-lang/Nim#19187
nim-lang/Nim#19179
nim-lang/Nim#19209
nim-lang/Nim#19210
nim-lang/Nim#19207
nim-lang/Nim#19219
nim-lang/Nim#19195
nim-lang/Nim#19212
nim-lang/Nim#19134
nim-lang/Nim#19235
nim-lang/Nim#19252
nim-lang/Nim#19196
nim-lang/Nim#19295
nim-lang/Nim#19301
nim-lang/Nim#19181
nim-lang/Nim#17223
nim-lang/Nim#19370
nim-lang/Nim#19385
nim-lang/Nim#19307
nim-lang/Nim#19394
nim-lang/Nim#19399
nim-lang/Nim#19390
nim-lang/Nim#19407
nim-lang/Nim#19419
nim-lang/Nim#19421
nim-lang/Nim#19363
nim-lang/Nim#19406
nim-lang/Nim#19431
nim-lang/Nim#19455
nim-lang/Nim#19461
nim-lang/Nim@cb894c7
nim-lang/Nim#19462
nim-lang/Nim#19442
nim-lang/Nim#19437
nim-lang/Nim#19433
nim-lang/Nim#19512
nim-lang/Nim#19487
nim-lang/Nim#19543

Excluded changes include major changes which require more consideration
and changes which don't apply to the current code anymore but could be
worth porting over still.

Excluded changes which only change the identifier casing in tests
or only concern code removed in nimskull aren't listed.

Commit listing is in the commit message

Co-authored-by: The Nim Contributors <>
a-mr added a commit to a-mr/Nim that referenced this pull request Dec 1, 2022
Fully implements nim-lang/RFCs#125
Follow-up of: nim-lang#18642 (for internal links)
and nim-lang#20127.

Overview
--------

Explicit import-like directive is required, called `.. importdoc::`.
(the syntax is % RST, Markdown will use it for a while).

Then one can reference any symbols/headings/anchors, as if they
were in the local file (but they will be prefixed with a module name
or markup document in link text).
It's possible to reference anything from anywhere (any direction
in `.nim`/`.md`/`.rst` files).

See `doc/docgen.md` for full description.

Working is based on `.idx` files, hence one needs to generate
all `.idx` beforehand. A dedicated option `--index:only` is introduced
(and a separate stage for `--index:only` is added to `kochdocs.nim`).

Performance note
----------------

Full run for `./koch docs` now takes 185% of the time before this PR.
(After: 315 s, before: 170 s on my PC).
All the time seems to be spent on `--index:only` run, which takes
almost as much (85%) of normal doc run -- it seems that most time
is spent on file parsing, turning off HTML generation phase has not
helped much.
(One could avoid it by specifying list of files that can be referenced
and pre-processing only them. But it can become error-prone and I assume
that these linke will be **everywhere** in the repository anyway,
especially considering nim-lang/RFCs#478.
So every `.nim`/`.md` file is processed for `.idx` first).

But that's all without significant part of repository converted to
cross-module auto links. To estimate impact I checked the time for
`doc`ing a few files (after all indexes have been generated), and
everywhere difference was **negligible**.
E.g. for `lib/std/private/osfiles.nim` that `importdoc`s large
`os.idx` and hence should have been a case with relatively large
performance impact, but:

* After: 0.59 s.
* Before: 0.59 s.

So Nim compiler works so slow that doc part basically does not matter :-)

Testing
-------

1) added `extlinks` test to `nimdoc/`
2) checked that `theindex.html` is still correct
2) fixed broken auto-links for modules that were derived from `os.nim`
   by adding appropriate ``importdoc``

Implementation note
-------------------

Parsing and formating of `.idx` entries is moved into a dedicated
`rstidx.nim` module from `rstgen.nim`.

`.idx` file format changed:

* fields are not escaped in most cases because we need original
  strings for referencing, not HTML ones
  (the exception is linkTitle for titles and headings).
  Escaping happens later -- on the stage of `rstgen` buildIndex, etc.
* all lines have fixed number of columns 6
* added discriminator tag as a first column,
  it always allows distinguish Nim/markup entries, titles/headings, etc.
  `rstgen` does not rely any more (in most cases) on ad-hoc logic
  to determine what type each entry is.
* there is now always a title entry added at the first line.
* add a line number as 6th column
* linkTitle (4th) column has a different format: before it was like
  `module: funcName()`, now it's `proc funcName()`.
  (This format is also propagated to `theindex.html` and search results,
  I kept it that way since I like it more though it's discussible.)
  This column is what used for Nim symbols resolution.
* also changed details on column format for headings and titles:
  "keyword" is original, "linkTitle" is HTML one
a-mr added a commit to a-mr/Nim that referenced this pull request Dec 16, 2022
Fully implements nim-lang/RFCs#125
Follow-up of: nim-lang#18642 (for internal links)
and nim-lang#20127.

Overview
--------

Explicit import-like directive is required, called `.. importdoc::`.
(the syntax is % RST, Markdown will use it for a while).

Then one can reference any symbols/headings/anchors, as if they
were in the local file (but they will be prefixed with a module name
or markup document in link text).
It's possible to reference anything from anywhere (any direction
in `.nim`/`.md`/`.rst` files).

See `doc/docgen.md` for full description.

Working is based on `.idx` files, hence one needs to generate
all `.idx` beforehand. A dedicated option `--index:only` is introduced
(and a separate stage for `--index:only` is added to `kochdocs.nim`).

Performance note
----------------

Full run for `./koch docs` now takes 185% of the time before this PR.
(After: 315 s, before: 170 s on my PC).
All the time seems to be spent on `--index:only` run, which takes
almost as much (85%) of normal doc run -- it seems that most time
is spent on file parsing, turning off HTML generation phase has not
helped much.
(One could avoid it by specifying list of files that can be referenced
and pre-processing only them. But it can become error-prone and I assume
that these linke will be **everywhere** in the repository anyway,
especially considering nim-lang/RFCs#478.
So every `.nim`/`.md` file is processed for `.idx` first).

But that's all without significant part of repository converted to
cross-module auto links. To estimate impact I checked the time for
`doc`ing a few files (after all indexes have been generated), and
everywhere difference was **negligible**.
E.g. for `lib/std/private/osfiles.nim` that `importdoc`s large
`os.idx` and hence should have been a case with relatively large
performance impact, but:

* After: 0.59 s.
* Before: 0.59 s.

So Nim compiler works so slow that doc part basically does not matter :-)

Testing
-------

1) added `extlinks` test to `nimdoc/`
2) checked that `theindex.html` is still correct
2) fixed broken auto-links for modules that were derived from `os.nim`
   by adding appropriate ``importdoc``

Implementation note
-------------------

Parsing and formating of `.idx` entries is moved into a dedicated
`rstidx.nim` module from `rstgen.nim`.

`.idx` file format changed:

* fields are not escaped in most cases because we need original
  strings for referencing, not HTML ones
  (the exception is linkTitle for titles and headings).
  Escaping happens later -- on the stage of `rstgen` buildIndex, etc.
* all lines have fixed number of columns 6
* added discriminator tag as a first column,
  it always allows distinguish Nim/markup entries, titles/headings, etc.
  `rstgen` does not rely any more (in most cases) on ad-hoc logic
  to determine what type each entry is.
* there is now always a title entry added at the first line.
* add a line number as 6th column
* linkTitle (4th) column has a different format: before it was like
  `module: funcName()`, now it's `proc funcName()`.
  (This format is also propagated to `theindex.html` and search results,
  I kept it that way since I like it more though it's discussible.)
  This column is what used for Nim symbols resolution.
* also changed details on column format for headings and titles:
  "keyword" is original, "linkTitle" is HTML one
Varriount pushed a commit that referenced this pull request Jan 4, 2023
* docgen: implement cross-document links

Fully implements nim-lang/RFCs#125
Follow-up of: #18642 (for internal links)
and #20127.

Overview
--------

Explicit import-like directive is required, called `.. importdoc::`.
(the syntax is % RST, Markdown will use it for a while).

Then one can reference any symbols/headings/anchors, as if they
were in the local file (but they will be prefixed with a module name
or markup document in link text).
It's possible to reference anything from anywhere (any direction
in `.nim`/`.md`/`.rst` files).

See `doc/docgen.md` for full description.

Working is based on `.idx` files, hence one needs to generate
all `.idx` beforehand. A dedicated option `--index:only` is introduced
(and a separate stage for `--index:only` is added to `kochdocs.nim`).

Performance note
----------------

Full run for `./koch docs` now takes 185% of the time before this PR.
(After: 315 s, before: 170 s on my PC).
All the time seems to be spent on `--index:only` run, which takes
almost as much (85%) of normal doc run -- it seems that most time
is spent on file parsing, turning off HTML generation phase has not
helped much.
(One could avoid it by specifying list of files that can be referenced
and pre-processing only them. But it can become error-prone and I assume
that these linke will be **everywhere** in the repository anyway,
especially considering nim-lang/RFCs#478.
So every `.nim`/`.md` file is processed for `.idx` first).

But that's all without significant part of repository converted to
cross-module auto links. To estimate impact I checked the time for
`doc`ing a few files (after all indexes have been generated), and
everywhere difference was **negligible**.
E.g. for `lib/std/private/osfiles.nim` that `importdoc`s large
`os.idx` and hence should have been a case with relatively large
performance impact, but:

* After: 0.59 s.
* Before: 0.59 s.

So Nim compiler works so slow that doc part basically does not matter :-)

Testing
-------

1) added `extlinks` test to `nimdoc/`
2) checked that `theindex.html` is still correct
2) fixed broken auto-links for modules that were derived from `os.nim`
   by adding appropriate ``importdoc``

Implementation note
-------------------

Parsing and formating of `.idx` entries is moved into a dedicated
`rstidx.nim` module from `rstgen.nim`.

`.idx` file format changed:

* fields are not escaped in most cases because we need original
  strings for referencing, not HTML ones
  (the exception is linkTitle for titles and headings).
  Escaping happens later -- on the stage of `rstgen` buildIndex, etc.
* all lines have fixed number of columns 6
* added discriminator tag as a first column,
  it always allows distinguish Nim/markup entries, titles/headings, etc.
  `rstgen` does not rely any more (in most cases) on ad-hoc logic
  to determine what type each entry is.
* there is now always a title entry added at the first line.
* add a line number as 6th column
* linkTitle (4th) column has a different format: before it was like
  `module: funcName()`, now it's `proc funcName()`.
  (This format is also propagated to `theindex.html` and search results,
  I kept it that way since I like it more though it's discussible.)
  This column is what used for Nim symbols resolution.
* also changed details on column format for headings and titles:
  "keyword" is original, "linkTitle" is HTML one

* fix paths on Windows + more clear code

* Update compiler/docgen.nim

Co-authored-by: Andreas Rumpf <[email protected]>

* Handle .md and .nim paths uniformly in findRefFile

* handle titles better + more comments

* don't allow markup overwrite index title for .nim files

Co-authored-by: Andreas Rumpf <[email protected]>
survivorm pushed a commit to survivorm/Nim that referenced this pull request Feb 28, 2023
* docgen: implement cross-document links

Fully implements nim-lang/RFCs#125
Follow-up of: nim-lang#18642 (for internal links)
and nim-lang#20127.

Overview
--------

Explicit import-like directive is required, called `.. importdoc::`.
(the syntax is % RST, Markdown will use it for a while).

Then one can reference any symbols/headings/anchors, as if they
were in the local file (but they will be prefixed with a module name
or markup document in link text).
It's possible to reference anything from anywhere (any direction
in `.nim`/`.md`/`.rst` files).

See `doc/docgen.md` for full description.

Working is based on `.idx` files, hence one needs to generate
all `.idx` beforehand. A dedicated option `--index:only` is introduced
(and a separate stage for `--index:only` is added to `kochdocs.nim`).

Performance note
----------------

Full run for `./koch docs` now takes 185% of the time before this PR.
(After: 315 s, before: 170 s on my PC).
All the time seems to be spent on `--index:only` run, which takes
almost as much (85%) of normal doc run -- it seems that most time
is spent on file parsing, turning off HTML generation phase has not
helped much.
(One could avoid it by specifying list of files that can be referenced
and pre-processing only them. But it can become error-prone and I assume
that these linke will be **everywhere** in the repository anyway,
especially considering nim-lang/RFCs#478.
So every `.nim`/`.md` file is processed for `.idx` first).

But that's all without significant part of repository converted to
cross-module auto links. To estimate impact I checked the time for
`doc`ing a few files (after all indexes have been generated), and
everywhere difference was **negligible**.
E.g. for `lib/std/private/osfiles.nim` that `importdoc`s large
`os.idx` and hence should have been a case with relatively large
performance impact, but:

* After: 0.59 s.
* Before: 0.59 s.

So Nim compiler works so slow that doc part basically does not matter :-)

Testing
-------

1) added `extlinks` test to `nimdoc/`
2) checked that `theindex.html` is still correct
2) fixed broken auto-links for modules that were derived from `os.nim`
   by adding appropriate ``importdoc``

Implementation note
-------------------

Parsing and formating of `.idx` entries is moved into a dedicated
`rstidx.nim` module from `rstgen.nim`.

`.idx` file format changed:

* fields are not escaped in most cases because we need original
  strings for referencing, not HTML ones
  (the exception is linkTitle for titles and headings).
  Escaping happens later -- on the stage of `rstgen` buildIndex, etc.
* all lines have fixed number of columns 6
* added discriminator tag as a first column,
  it always allows distinguish Nim/markup entries, titles/headings, etc.
  `rstgen` does not rely any more (in most cases) on ad-hoc logic
  to determine what type each entry is.
* there is now always a title entry added at the first line.
* add a line number as 6th column
* linkTitle (4th) column has a different format: before it was like
  `module: funcName()`, now it's `proc funcName()`.
  (This format is also propagated to `theindex.html` and search results,
  I kept it that way since I like it more though it's discussible.)
  This column is what used for Nim symbols resolution.
* also changed details on column format for headings and titles:
  "keyword" is original, "linkTitle" is HTML one

* fix paths on Windows + more clear code

* Update compiler/docgen.nim

Co-authored-by: Andreas Rumpf <[email protected]>

* Handle .md and .nim paths uniformly in findRefFile

* handle titles better + more comments

* don't allow markup overwrite index title for .nim files

Co-authored-by: Andreas Rumpf <[email protected]>
capocasa pushed a commit to capocasa/Nim that referenced this pull request Mar 31, 2023
* docgen: implement cross-document links

Fully implements nim-lang/RFCs#125
Follow-up of: nim-lang#18642 (for internal links)
and nim-lang#20127.

Overview
--------

Explicit import-like directive is required, called `.. importdoc::`.
(the syntax is % RST, Markdown will use it for a while).

Then one can reference any symbols/headings/anchors, as if they
were in the local file (but they will be prefixed with a module name
or markup document in link text).
It's possible to reference anything from anywhere (any direction
in `.nim`/`.md`/`.rst` files).

See `doc/docgen.md` for full description.

Working is based on `.idx` files, hence one needs to generate
all `.idx` beforehand. A dedicated option `--index:only` is introduced
(and a separate stage for `--index:only` is added to `kochdocs.nim`).

Performance note
----------------

Full run for `./koch docs` now takes 185% of the time before this PR.
(After: 315 s, before: 170 s on my PC).
All the time seems to be spent on `--index:only` run, which takes
almost as much (85%) of normal doc run -- it seems that most time
is spent on file parsing, turning off HTML generation phase has not
helped much.
(One could avoid it by specifying list of files that can be referenced
and pre-processing only them. But it can become error-prone and I assume
that these linke will be **everywhere** in the repository anyway,
especially considering nim-lang/RFCs#478.
So every `.nim`/`.md` file is processed for `.idx` first).

But that's all without significant part of repository converted to
cross-module auto links. To estimate impact I checked the time for
`doc`ing a few files (after all indexes have been generated), and
everywhere difference was **negligible**.
E.g. for `lib/std/private/osfiles.nim` that `importdoc`s large
`os.idx` and hence should have been a case with relatively large
performance impact, but:

* After: 0.59 s.
* Before: 0.59 s.

So Nim compiler works so slow that doc part basically does not matter :-)

Testing
-------

1) added `extlinks` test to `nimdoc/`
2) checked that `theindex.html` is still correct
2) fixed broken auto-links for modules that were derived from `os.nim`
   by adding appropriate ``importdoc``

Implementation note
-------------------

Parsing and formating of `.idx` entries is moved into a dedicated
`rstidx.nim` module from `rstgen.nim`.

`.idx` file format changed:

* fields are not escaped in most cases because we need original
  strings for referencing, not HTML ones
  (the exception is linkTitle for titles and headings).
  Escaping happens later -- on the stage of `rstgen` buildIndex, etc.
* all lines have fixed number of columns 6
* added discriminator tag as a first column,
  it always allows distinguish Nim/markup entries, titles/headings, etc.
  `rstgen` does not rely any more (in most cases) on ad-hoc logic
  to determine what type each entry is.
* there is now always a title entry added at the first line.
* add a line number as 6th column
* linkTitle (4th) column has a different format: before it was like
  `module: funcName()`, now it's `proc funcName()`.
  (This format is also propagated to `theindex.html` and search results,
  I kept it that way since I like it more though it's discussible.)
  This column is what used for Nim symbols resolution.
* also changed details on column format for headings and titles:
  "keyword" is original, "linkTitle" is HTML one

* fix paths on Windows + more clear code

* Update compiler/docgen.nim

Co-authored-by: Andreas Rumpf <[email protected]>

* Handle .md and .nim paths uniformly in findRefFile

* handle titles better + more comments

* don't allow markup overwrite index title for .nim files

Co-authored-by: Andreas Rumpf <[email protected]>
bung87 pushed a commit to bung87/Nim that referenced this pull request Jul 29, 2023
* docgen: implement cross-document links

Fully implements nim-lang/RFCs#125
Follow-up of: nim-lang#18642 (for internal links)
and nim-lang#20127.

Overview
--------

Explicit import-like directive is required, called `.. importdoc::`.
(the syntax is % RST, Markdown will use it for a while).

Then one can reference any symbols/headings/anchors, as if they
were in the local file (but they will be prefixed with a module name
or markup document in link text).
It's possible to reference anything from anywhere (any direction
in `.nim`/`.md`/`.rst` files).

See `doc/docgen.md` for full description.

Working is based on `.idx` files, hence one needs to generate
all `.idx` beforehand. A dedicated option `--index:only` is introduced
(and a separate stage for `--index:only` is added to `kochdocs.nim`).

Performance note
----------------

Full run for `./koch docs` now takes 185% of the time before this PR.
(After: 315 s, before: 170 s on my PC).
All the time seems to be spent on `--index:only` run, which takes
almost as much (85%) of normal doc run -- it seems that most time
is spent on file parsing, turning off HTML generation phase has not
helped much.
(One could avoid it by specifying list of files that can be referenced
and pre-processing only them. But it can become error-prone and I assume
that these linke will be **everywhere** in the repository anyway,
especially considering nim-lang/RFCs#478.
So every `.nim`/`.md` file is processed for `.idx` first).

But that's all without significant part of repository converted to
cross-module auto links. To estimate impact I checked the time for
`doc`ing a few files (after all indexes have been generated), and
everywhere difference was **negligible**.
E.g. for `lib/std/private/osfiles.nim` that `importdoc`s large
`os.idx` and hence should have been a case with relatively large
performance impact, but:

* After: 0.59 s.
* Before: 0.59 s.

So Nim compiler works so slow that doc part basically does not matter :-)

Testing
-------

1) added `extlinks` test to `nimdoc/`
2) checked that `theindex.html` is still correct
2) fixed broken auto-links for modules that were derived from `os.nim`
   by adding appropriate ``importdoc``

Implementation note
-------------------

Parsing and formating of `.idx` entries is moved into a dedicated
`rstidx.nim` module from `rstgen.nim`.

`.idx` file format changed:

* fields are not escaped in most cases because we need original
  strings for referencing, not HTML ones
  (the exception is linkTitle for titles and headings).
  Escaping happens later -- on the stage of `rstgen` buildIndex, etc.
* all lines have fixed number of columns 6
* added discriminator tag as a first column,
  it always allows distinguish Nim/markup entries, titles/headings, etc.
  `rstgen` does not rely any more (in most cases) on ad-hoc logic
  to determine what type each entry is.
* there is now always a title entry added at the first line.
* add a line number as 6th column
* linkTitle (4th) column has a different format: before it was like
  `module: funcName()`, now it's `proc funcName()`.
  (This format is also propagated to `theindex.html` and search results,
  I kept it that way since I like it more though it's discussible.)
  This column is what used for Nim symbols resolution.
* also changed details on column format for headings and titles:
  "keyword" is original, "linkTitle" is HTML one

* fix paths on Windows + more clear code

* Update compiler/docgen.nim

Co-authored-by: Andreas Rumpf <[email protected]>

* Handle .md and .nim paths uniformly in findRefFile

* handle titles better + more comments

* don't allow markup overwrite index title for .nim files

Co-authored-by: Andreas Rumpf <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants